Sep 11 11:28:22.410923 ip-10-0-142-17 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 11:28:22.410938 ip-10-0-142-17 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 11:28:22.410948 ip-10-0-142-17 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 11:28:22.411408 ip-10-0-142-17 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 11:28:32.656533 ip-10-0-142-17 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 11:28:32.656552 ip-10-0-142-17 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 39b4ca6cfe91472290f2685387135711 -- Sep 11 11:30:23.363047 ip-10-0-142-17 systemd[1]: Starting Kubernetes Kubelet... Sep 11 11:30:23.710002 ip-10-0-142-17 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 11:30:23.710002 ip-10-0-142-17 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 11:30:23.710002 ip-10-0-142-17 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 11:30:23.710002 ip-10-0-142-17 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 11:30:23.710002 ip-10-0-142-17 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 11:30:23.710835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.710090 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 11:30:23.713087 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713065 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:30:23.713087 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713085 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:30:23.713087 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713090 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713094 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713098 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713102 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713106 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713110 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713114 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713117 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713121 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713125 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713129 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713133 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713136 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713141 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713145 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713148 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713152 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713156 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713160 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713165 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:30:23.713345 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713169 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713173 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713177 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713192 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713196 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713200 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713204 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713208 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713212 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713216 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713220 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713224 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713228 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713232 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713238 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713243 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713247 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713251 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713255 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:30:23.714199 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713263 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713269 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713274 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713279 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713283 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713287 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713293 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713298 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713303 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713308 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713313 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713317 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713321 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713325 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713329 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713334 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713338 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713342 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713346 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713349 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:30:23.715019 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713353 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713357 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713363 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713369 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713373 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713378 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713382 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713387 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713392 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713399 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713403 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713408 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713412 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713416 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713420 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713425 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713429 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713433 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713437 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:30:23.715867 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713442 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713446 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713451 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713456 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713460 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713464 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713468 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713472 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713476 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713483 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713488 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713494 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713499 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713503 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713507 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713511 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713515 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713520 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.713524 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:30:23.716670 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714683 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714695 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714701 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714705 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714710 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714738 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714743 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714747 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714752 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714756 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714760 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714764 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714768 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714772 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714778 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714782 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714787 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714791 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714795 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714799 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:30:23.717531 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714804 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714808 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714812 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714816 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714825 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714830 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714835 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714840 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714844 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714848 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714853 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714859 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714863 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714868 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714872 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714876 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714880 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714884 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714888 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:30:23.718320 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714892 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714896 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714900 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714904 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714908 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714913 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714917 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714921 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714927 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714930 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714934 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714939 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714943 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714947 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714951 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714955 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714959 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714963 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714968 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714972 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:30:23.718857 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714976 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714980 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714984 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714988 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714993 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.714997 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715002 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715006 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715010 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715014 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715018 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715022 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715026 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715030 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715034 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715038 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715041 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715046 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715050 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715054 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:30:23.719445 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715058 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715063 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715068 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715072 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715076 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715080 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715084 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715088 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715092 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715096 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715104 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715110 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715116 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715120 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715124 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715128 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715132 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715136 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715141 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715145 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:30:23.720124 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715833 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715843 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715849 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715853 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715858 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715864 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715868 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715872 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715876 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715880 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715884 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715889 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715893 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715898 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715903 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715907 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715912 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715916 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715920 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715924 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:30:23.720739 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715929 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715935 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715939 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715943 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715947 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715951 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715956 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715960 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715964 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715970 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715975 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715979 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715983 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715987 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715991 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.715995 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716000 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716006 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716012 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716017 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:30:23.721263 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716021 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716025 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716030 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716034 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716039 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716043 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716048 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716052 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716058 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716064 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716069 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716075 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716079 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716084 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716088 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716092 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716096 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716100 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716104 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716108 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:30:23.721881 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716112 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716116 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716120 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716124 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716128 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716132 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716136 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716141 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716145 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716150 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716153 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716158 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716164 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716170 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716175 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716179 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716185 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716188 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716193 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716198 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:30:23.722375 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716202 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716206 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716210 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716214 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716218 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716223 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716227 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716231 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716235 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716239 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716243 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716247 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716251 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716255 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716259 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716263 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716267 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716272 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.716276 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716367 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 11:30:23.722914 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716420 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716431 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716437 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716443 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716448 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716454 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716460 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716465 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716476 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716481 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716486 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716491 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716497 2579 flags.go:64] FLAG: --cgroup-root="" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716501 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716506 2579 flags.go:64] FLAG: --client-ca-file="" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716511 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716515 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716524 2579 flags.go:64] FLAG: --cluster-domain="" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716529 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716534 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716539 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716544 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716550 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 11:30:23.723408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716555 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716560 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716566 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716571 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716575 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716580 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716584 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716589 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716596 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716601 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716605 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716609 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716614 2579 flags.go:64] FLAG: --enable-server="true" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716619 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716705 2579 flags.go:64] FLAG: --event-burst="100" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716711 2579 flags.go:64] FLAG: --event-qps="50" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716737 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716744 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716748 2579 flags.go:64] FLAG: --eviction-hard="" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716755 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716759 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716764 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716770 2579 flags.go:64] FLAG: --eviction-soft="" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716774 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716779 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 11:30:23.723983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716784 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716790 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716795 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716800 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716804 2579 flags.go:64] FLAG: --feature-gates="" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716810 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716815 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716820 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716825 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716830 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716834 2579 flags.go:64] FLAG: --help="false" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716839 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-17.ec2.internal" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716844 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716848 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716853 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716859 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716865 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716870 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716875 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716879 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716884 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716888 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716893 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716898 2579 flags.go:64] FLAG: --kube-reserved="" Sep 11 11:30:23.724650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716905 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716909 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716914 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716918 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716923 2579 flags.go:64] FLAG: --lock-file="" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716927 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716932 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716938 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716946 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716952 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716957 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716961 2579 flags.go:64] FLAG: --logging-format="text" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716967 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716971 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716976 2579 flags.go:64] FLAG: --manifest-url="" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716980 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716987 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716992 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.716998 2579 flags.go:64] FLAG: --max-pods="110" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717003 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717007 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717012 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717016 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717021 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717026 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 11:30:23.725324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717031 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717041 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717046 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717051 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717056 2579 flags.go:64] FLAG: --pod-cidr="" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717061 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717070 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717077 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717081 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717086 2579 flags.go:64] FLAG: --port="10250" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717091 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717096 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f88dbaa27b066bcd" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717101 2579 flags.go:64] FLAG: --qos-reserved="" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717106 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717111 2579 flags.go:64] FLAG: --register-node="true" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717116 2579 flags.go:64] FLAG: --register-with-taints="" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717122 2579 flags.go:64] FLAG: --registry-burst="10" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717128 2579 flags.go:64] FLAG: --registry-qps="5" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717133 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717137 2579 flags.go:64] FLAG: --reserved-memory="" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717143 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717148 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717153 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717157 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717162 2579 flags.go:64] FLAG: --runonce="false" Sep 11 11:30:23.725965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717167 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717172 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717177 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717181 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717185 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717190 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717196 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717201 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717206 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717210 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717215 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717219 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717224 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717229 2579 flags.go:64] FLAG: --system-cgroups="" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717235 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717244 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717248 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717253 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717259 2579 flags.go:64] FLAG: --tls-min-version="" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717264 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717268 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717273 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717278 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717283 2579 flags.go:64] FLAG: --v="2" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717291 2579 flags.go:64] FLAG: --version="false" Sep 11 11:30:23.726562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717297 2579 flags.go:64] FLAG: --vmodule="" Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717303 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717308 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717505 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717512 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717517 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717521 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717526 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717530 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717534 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717539 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717544 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717549 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717553 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717557 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717562 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717566 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717570 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717574 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717578 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:30:23.727194 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717582 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717588 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717593 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717597 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717602 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717606 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717610 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717614 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717618 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717622 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717627 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717634 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717638 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717642 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717646 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717650 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717657 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717672 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717677 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:30:23.727671 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717682 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717687 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717691 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717695 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717700 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717704 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717708 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717729 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717736 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717742 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717747 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717751 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717755 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717759 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717765 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717770 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717774 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717779 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717783 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717787 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:30:23.728159 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717791 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717795 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717799 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717803 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717810 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717815 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717819 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717823 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717827 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717831 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717835 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717839 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717843 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717847 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717851 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717856 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717860 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717865 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717868 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717873 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:30:23.728648 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717877 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717881 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717885 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717890 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717894 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717898 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717904 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717908 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717912 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717916 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717920 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717926 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717932 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717938 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717942 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717946 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717951 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717957 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717961 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:30:23.729195 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717965 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717969 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717973 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.717977 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.717990 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.726158 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.726173 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.726186 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.726195 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726251 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726256 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726260 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726263 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726266 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:30:23.729684 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726269 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726273 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726277 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726280 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726283 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726285 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726288 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726290 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726293 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726296 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726298 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726301 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726303 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726306 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726309 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726312 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726317 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726320 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726323 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:30:23.730060 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726326 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726329 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726331 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726334 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726337 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726340 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726343 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726345 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726348 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726351 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726353 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726356 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726359 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726361 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726364 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726366 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726369 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726371 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726374 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726376 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:30:23.730519 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726379 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726381 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726384 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726386 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726389 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726391 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726393 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726396 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726398 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726401 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726404 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726406 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726409 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726412 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726415 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726417 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726419 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726422 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726425 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:30:23.731078 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726427 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726431 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726434 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726437 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726440 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726443 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726446 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726449 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726451 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726454 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726457 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726459 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726462 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726465 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726468 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726471 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726474 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726476 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726478 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726481 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:30:23.731556 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726483 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726486 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726488 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726491 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726497 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726499 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726502 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726505 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726507 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726510 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726513 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726516 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726518 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726521 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726523 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:30:23.732081 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726526 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.726530 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726655 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726660 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726663 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726666 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726668 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726671 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726673 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726676 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726679 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726683 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726685 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726688 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726690 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:30:23.732478 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726693 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726695 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726697 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726700 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726703 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726705 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726708 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726711 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726731 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726734 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726737 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726740 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726742 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726745 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726748 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726750 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726753 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726755 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726758 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726760 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:30:23.732888 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726763 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726765 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726768 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726770 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726773 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726775 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726777 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726780 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726782 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726784 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726787 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726789 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726792 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726794 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726797 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726799 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726801 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726804 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726806 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726809 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:30:23.733414 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726812 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726815 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726818 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726820 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726822 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726825 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726828 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726830 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726832 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726847 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726852 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726855 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726857 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726860 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726863 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726866 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726868 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726870 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726873 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726875 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:30:23.733953 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726878 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726881 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726883 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726886 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726888 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726891 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726893 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726896 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726898 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726901 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726903 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726905 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726908 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726910 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726914 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726918 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726920 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726924 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726927 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:30:23.734486 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726929 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726932 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726935 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726938 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726941 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726944 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:23.726947 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.726952 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.727042 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.729390 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.730451 2579 server.go:1019] "Starting client certificate rotation" Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.730550 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.730588 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 11:30:23.735122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.732687 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 11:30:23.735947 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.735926 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 11:30:23.744521 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.744507 2579 log.go:25] "Validated CRI v1 runtime API" Sep 11 11:30:23.747351 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.747337 2579 log.go:25] "Validated CRI v1 image API" Sep 11 11:30:23.748243 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.748227 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 11:30:23.751807 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.751781 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 85bc2a57-8273-498e-a485-91b6e4b9be2e:/dev/nvme0n1p3 8de0f971-45d3-4eaa-b943-73d7e2229ee4:/dev/nvme0n1p4] Sep 11 11:30:23.751880 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.751807 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 11:30:23.756207 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.756097 2579 manager.go:216] Machine: {Timestamp:2026-09-11 11:30:23.754936857 +0000 UTC m=+0.290843844 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094136 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a3cb86ef505be6fa383fb3542e51 SystemUUID:ec28a3cb-86ef-505b-e6fa-383fb3542e51 BootID:39b4ca6c-fe91-4722-90f2-685387135711 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:42:74:06:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:42:74:06:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:4c:b4:6b:13:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 11:30:23.756207 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.756199 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 11:30:23.756351 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.756299 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 11:30:23.759303 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.759278 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 11:30:23.759422 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.759305 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 11:30:23.760558 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.760548 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 11:30:23.760591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.760560 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 11:30:23.760591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.760569 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 11:30:23.761156 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.761147 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 11:30:23.762087 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.762076 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 11:30:23.766000 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.765989 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 11:30:23.766115 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766105 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 11:30:23.766143 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766134 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 11 11:30:23.766182 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766144 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 11:30:23.766182 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766157 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 11:30:23.766182 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766166 2579 kubelet.go:398] "Adding apiserver pod source" Sep 11 11:30:23.766182 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766174 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 11:30:23.766509 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766491 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 11:30:23.766591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766578 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 11:30:23.766634 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.766603 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 11:30:23.769502 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.769485 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 11:30:23.772073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.772031 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 11:30:23.772167 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.772081 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 11:30:23.774871 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774815 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 11:30:23.774871 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774847 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 11:30:23.774871 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774854 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 11:30:23.774871 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774860 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 11:30:23.774871 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774869 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 11:30:23.774871 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774875 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 11:30:23.775122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774889 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 11:30:23.775122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774894 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 11:30:23.775122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774901 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 11:30:23.775122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774906 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 11:30:23.775122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774914 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 11:30:23.775122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.774919 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 11:30:23.775422 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.775403 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 11:30:23.775493 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.775477 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 11:30:23.775571 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.775562 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 11:30:23.775599 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.775571 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 11:30:23.778149 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.778134 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 11:30:23.779172 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.779161 2579 server.go:1268] "Started kubelet" Sep 11 11:30:23.779271 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.779248 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 11:30:23.779383 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.779260 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 11:30:23.779383 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.779290 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 11:30:23.780008 ip-10-0-142-17 systemd[1]: Started Kubernetes Kubelet. Sep 11 11:30:23.780498 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.780441 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 11:30:23.781127 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.781113 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 11 11:30:23.784832 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.784814 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 11:30:23.786627 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.786613 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 11:30:23.786627 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.786619 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 11:30:23.786774 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.786751 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:23.787408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787388 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 11:30:23.787489 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787455 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 11:30:23.787489 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787472 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 11:30:23.787596 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787532 2579 factory.go:55] Registering systemd factory Sep 11 11:30:23.787596 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787546 2579 factory.go:223] Registration of the systemd container factory successfully Sep 11 11:30:23.787596 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787578 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 11 11:30:23.787596 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787587 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 11 11:30:23.787785 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787775 2579 factory.go:153] Registering CRI-O factory Sep 11 11:30:23.787831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787789 2579 factory.go:223] Registration of the crio container factory successfully Sep 11 11:30:23.787887 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787854 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 11:30:23.787887 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787873 2579 factory.go:103] Registering Raw factory Sep 11 11:30:23.787887 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.787888 2579 manager.go:1196] Started watching for new ooms in manager Sep 11 11:30:23.788217 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.788204 2579 manager.go:349] Starting recovery of all containers Sep 11 11:30:23.792638 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.792607 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 11:30:23.792997 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.792968 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 11:30:23.793376 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.793355 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nnmhq" Sep 11 11:30:23.794095 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.793135 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-17.ec2.internal.18d440d242b0fcde default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-17.ec2.internal,UID:ip-10-0-142-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-17.ec2.internal,},FirstTimestamp:2026-09-11 11:30:23.779134686 +0000 UTC m=+0.315041674,LastTimestamp:2026-09-11 11:30:23.779134686 +0000 UTC m=+0.315041674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-17.ec2.internal,}" Sep 11 11:30:23.796969 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.796828 2579 manager.go:354] Recovery completed Sep 11 11:30:23.803494 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.803472 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nnmhq" Sep 11 11:30:23.803941 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.803929 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:30:23.805673 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.805650 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:30:23.805745 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.805687 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:30:23.805745 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.805697 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:30:23.806210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.806196 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 11:30:23.806210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.806207 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 11:30:23.806318 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.806212 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:30:23.806318 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.806223 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 11:30:23.806935 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.806917 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:30:23.807004 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.806946 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:30:23.807004 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.806958 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:30:23.808037 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.807975 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-17.ec2.internal.18d440d24445eba9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-17.ec2.internal,UID:ip-10-0-142-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-17.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-17.ec2.internal,},FirstTimestamp:2026-09-11 11:30:23.805672361 +0000 UTC m=+0.341579349,LastTimestamp:2026-09-11 11:30:23.805672361 +0000 UTC m=+0.341579349,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-17.ec2.internal,}" Sep 11 11:30:23.808192 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.808179 2579 policy_none.go:49] "None policy: Start" Sep 11 11:30:23.808192 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.808194 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 11:30:23.808294 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.808204 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 11:30:23.810172 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.810160 2579 policy_none.go:47] "Start" Sep 11 11:30:23.859279 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.859264 2579 manager.go:337] "Starting Device Plugin manager" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.859298 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.859308 2579 server.go:87] "Starting device plugin registration server" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.859579 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.859589 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.859686 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.859785 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.859793 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.860270 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 11:30:23.872759 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.860317 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:23.892461 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.892439 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 11:30:23.893757 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.893740 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 11:30:23.893810 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.893796 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 11:30:23.893850 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.893823 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 11:30:23.893926 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.893894 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 11:30:23.896263 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.896246 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:30:23.960306 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.960250 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:30:23.961071 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.961057 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:30:23.961130 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.961087 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:30:23.961130 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.961103 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:30:23.961190 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.961132 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-17.ec2.internal" Sep 11 11:30:23.967375 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.967362 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-17.ec2.internal" Sep 11 11:30:23.967415 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.967381 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-17.ec2.internal\": node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:23.985904 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:23.985889 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:23.994335 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.994311 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal"] Sep 11 11:30:23.994379 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.994368 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:30:23.995652 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.995630 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:30:23.995764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.995674 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:30:23.995764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.995685 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:30:23.998022 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998007 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:30:23.998153 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" Sep 11 11:30:23.998211 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998172 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:30:23.998739 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998705 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:30:23.998835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998750 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:30:23.998835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998706 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:30:23.998835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998760 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:30:23.998835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998783 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:30:23.998835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:23.998800 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:30:24.001111 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.001095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.001191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.001124 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:30:24.001811 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.001795 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:30:24.001879 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.001826 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:30:24.001879 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.001850 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:30:24.022279 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.022262 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-17.ec2.internal\" not found" node="ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.026636 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.026622 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-17.ec2.internal\" not found" node="ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.086521 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.086501 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.088765 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.088749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4132a9a58a5c237803951588c311249-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal\" (UID: \"e4132a9a58a5c237803951588c311249\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.088827 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.088772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4132a9a58a5c237803951588c311249-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal\" (UID: \"e4132a9a58a5c237803951588c311249\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.088827 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.088787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c14bf53574a5faf7f92e569f4d21b76b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-17.ec2.internal\" (UID: \"c14bf53574a5faf7f92e569f4d21b76b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.186765 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.186743 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.188995 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.188980 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4132a9a58a5c237803951588c311249-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal\" (UID: \"e4132a9a58a5c237803951588c311249\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.189050 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.189006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4132a9a58a5c237803951588c311249-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal\" (UID: \"e4132a9a58a5c237803951588c311249\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.189085 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.189053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c14bf53574a5faf7f92e569f4d21b76b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-17.ec2.internal\" (UID: \"c14bf53574a5faf7f92e569f4d21b76b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.189085 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.189069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4132a9a58a5c237803951588c311249-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal\" (UID: \"e4132a9a58a5c237803951588c311249\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.189156 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.189095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4132a9a58a5c237803951588c311249-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal\" (UID: \"e4132a9a58a5c237803951588c311249\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.189156 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.189140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c14bf53574a5faf7f92e569f4d21b76b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-17.ec2.internal\" (UID: \"c14bf53574a5faf7f92e569f4d21b76b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.287383 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.287343 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.325655 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.325628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.329276 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.329260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.388324 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.388301 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.488681 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.488665 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.589225 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.589172 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.689755 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.689729 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.730031 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.730007 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 11:30:24.730640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.730156 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 11:30:24.789843 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.789822 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.807743 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.807695 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 11:25:24 +0000 UTC" deadline="2028-03-12 22:44:43.472258301 +0000 UTC" Sep 11 11:30:24.807743 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.807739 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13163h14m18.66452272s" Sep 11 11:30:24.851112 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:24.851077 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4132a9a58a5c237803951588c311249.slice/crio-c457377a236ae0a215e1f8eef092c0ac66c1c12399eef12dfdceb11b60b7e2d5 WatchSource:0}: Error finding container c457377a236ae0a215e1f8eef092c0ac66c1c12399eef12dfdceb11b60b7e2d5: Status 404 returned error can't find the container with id c457377a236ae0a215e1f8eef092c0ac66c1c12399eef12dfdceb11b60b7e2d5 Sep 11 11:30:24.851489 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:24.851472 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14bf53574a5faf7f92e569f4d21b76b.slice/crio-6351dc1dadaeefadb4616bb30bc0a007fae4aede324396ffec203be28889a21a WatchSource:0}: Error finding container 6351dc1dadaeefadb4616bb30bc0a007fae4aede324396ffec203be28889a21a: Status 404 returned error can't find the container with id 6351dc1dadaeefadb4616bb30bc0a007fae4aede324396ffec203be28889a21a Sep 11 11:30:24.857325 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.857310 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:30:24.890364 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:24.890339 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-17.ec2.internal\" not found" Sep 11 11:30:24.896573 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.896523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" event={"ID":"c14bf53574a5faf7f92e569f4d21b76b","Type":"ContainerStarted","Data":"6351dc1dadaeefadb4616bb30bc0a007fae4aede324396ffec203be28889a21a"} Sep 11 11:30:24.897639 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.897619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" event={"ID":"e4132a9a58a5c237803951588c311249","Type":"ContainerStarted","Data":"c457377a236ae0a215e1f8eef092c0ac66c1c12399eef12dfdceb11b60b7e2d5"} Sep 11 11:30:24.969027 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.969003 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:30:24.987289 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.987266 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" Sep 11 11:30:24.988640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.988627 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:30:24.999504 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:24.999484 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 11:30:25.001332 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.001320 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" Sep 11 11:30:25.010829 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.010811 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 11:30:25.370848 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.370825 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:30:25.768363 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.768318 2579 apiserver.go:52] "Watching apiserver" Sep 11 11:30:25.778294 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.777362 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 11:30:25.778294 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.777929 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal","openshift-multus/multus-7j8x7","openshift-multus/multus-additional-cni-plugins-8kkp9","openshift-network-diagnostics/network-check-target-prvf7","openshift-network-operator/iptables-alerter-lp96t","kube-system/global-pull-secret-syncer-d6f7p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84","openshift-cluster-node-tuning-operator/tuned-k5mxn","openshift-dns/node-resolver-fzqcl","openshift-image-registry/node-ca-4s48h","openshift-multus/network-metrics-daemon-59gwj","openshift-ovn-kubernetes/ovnkube-node-zcd47","kube-system/konnectivity-agent-dxpvh"] Sep 11 11:30:25.780772 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.780522 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.780772 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:25.780611 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:25.784002 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.783082 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.785460 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.785434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.786861 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.786835 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52nxz\"" Sep 11 11:30:25.786861 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.786853 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 11:30:25.787039 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.786903 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 11:30:25.787119 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.787060 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 11:30:25.787264 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.787243 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 11:30:25.787342 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.787295 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 11:30:25.787545 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.787255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 11:30:25.788447 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.787723 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 11:30:25.788447 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.787978 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 11:30:25.788447 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.788103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.788447 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.788343 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 11:30:25.788708 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.788528 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 11:30:25.788765 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.788720 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dfqw9\"" Sep 11 11:30:25.788921 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.788905 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 11:30:25.791048 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.790599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:25.791048 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.790646 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 11:30:25.791048 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:25.790669 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:25.791048 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.790712 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5khzm\"" Sep 11 11:30:25.791048 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.791022 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 11:30:25.792977 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.792816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:25.795277 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.795255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:30:25.795631 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.795618 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 11:30:25.795849 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.795830 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 11:30:25.796097 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.796081 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2895m\"" Sep 11 11:30:25.798377 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.798355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.799740 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.799718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:25.800248 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-run-ovn-kubernetes\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800248 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-netns\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.800248 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-etc-kubernetes\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.800248 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtnp\" (UniqueName: \"kubernetes.io/projected/e09db19d-ecec-41b5-85e3-164979161e08-kube-api-access-9rtnp\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.800476 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-systemd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800476 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-var-lib-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800476 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-ovn\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800476 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800454 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 11:30:25.800476 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-cni-binary-copy\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.800756 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-k8s-cni-cncf-io\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.800756 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.800756 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.800934 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-kubelet\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800934 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-run-netns\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800934 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-cni-bin\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800934 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-ovnkube-config\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.800934 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-cni-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-hostroot\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.800970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-multus-certs\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-system-cni-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-cnibin\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-conf-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-daemon-config\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-system-cni-dir\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-cni-binary-copy\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.801275 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-slash\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-etc-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77a32e00-3aac-4894-97c2-42327fc00c08-ovn-node-metrics-cert\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-cni-bin\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-cni-multus\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qhk\" (UniqueName: \"kubernetes.io/projected/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-kube-api-access-w9qhk\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-cnibin\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/222b1b29-9ac9-4418-b6c2-271ff86e4850-kubelet-config\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-env-overrides\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-ovnkube-script-lib\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.801744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-socket-dir-parent\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-systemd-units\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.801897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-node-log\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-cni-netd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrrd\" (UniqueName: \"kubernetes.io/projected/77a32e00-3aac-4894-97c2-42327fc00c08-kube-api-access-nvrrd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802178 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lkpqp\"" Sep 11 11:30:25.802274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-kubelet\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.802688 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/222b1b29-9ac9-4418-b6c2-271ff86e4850-dbus\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.802688 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-log-socket\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.802688 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802555 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 11:30:25.802688 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802664 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 11:30:25.802688 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.802668 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-khwbq\"" Sep 11 11:30:25.803075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.803056 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.803718 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.803701 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 11:30:25.803775 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.803756 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:30:25.807315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.806417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:25.807315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.806665 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 11:30:25.807315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.806912 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 11:30:25.807315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.806956 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5pk8d\"" Sep 11 11:30:25.807315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.807150 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 11:30:25.808906 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.808874 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfgnl\"" Sep 11 11:30:25.809227 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.809206 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 11:30:25.809316 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.809241 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 11:30:25.809521 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.809498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:25.812712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.811762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:25.812712 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:25.811833 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:25.812712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.811975 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rqzjv\"" Sep 11 11:30:25.812712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.812142 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 11:30:25.812712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.812250 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 11:30:25.812712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.812500 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 11:30:25.829025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.829006 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wbpcs" Sep 11 11:30:25.839055 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.839032 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wbpcs" Sep 11 11:30:25.889119 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.889091 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 11:30:25.902927 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.902903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-node-log\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.902940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrrd\" (UniqueName: \"kubernetes.io/projected/77a32e00-3aac-4894-97c2-42327fc00c08-kube-api-access-nvrrd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.902971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4brc\" (UniqueName: \"kubernetes.io/projected/7eafd60c-bb2d-4473-ab8a-76180f836548-kube-api-access-n4brc\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:25.903054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.902999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-etc-selinux\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.903054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/222b1b29-9ac9-4418-b6c2-271ff86e4850-dbus\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-netns\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-etc-kubernetes\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtnp\" (UniqueName: \"kubernetes.io/projected/e09db19d-ecec-41b5-85e3-164979161e08-kube-api-access-9rtnp\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-tmp\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-systemd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-var-lib-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-ovn\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvm7c\" (UniqueName: \"kubernetes.io/projected/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-kube-api-access-pvm7c\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:25.903317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-iptables-alerter-script\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-run-netns\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-cni-bin\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-cni-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-systemd\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-var-lib-kubelet\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-system-cni-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-cni-binary-copy\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncg7d\" (UniqueName: \"kubernetes.io/projected/3499a298-490e-489b-a5fb-307cd8fcf6e9-kube-api-access-ncg7d\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-slash\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qhk\" (UniqueName: \"kubernetes.io/projected/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-kube-api-access-w9qhk\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-lib-modules\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.903901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-ovnkube-script-lib\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysctl-d\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-run\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-tuned\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0f63ea2-446e-4fdc-9da3-d28ac08c3727-konnectivity-ca\") pod \"konnectivity-agent-dxpvh\" (UID: \"a0f63ea2-446e-4fdc-9da3-d28ac08c3727\") " pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-systemd-units\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.903992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-cni-netd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-kubelet\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpb9\" (UniqueName: \"kubernetes.io/projected/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-kube-api-access-8tpb9\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-log-socket\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-run-ovn-kubernetes\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-kubernetes\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysctl-conf\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-sys-fs\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-cni-binary-copy\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-k8s-cni-cncf-io\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0f63ea2-446e-4fdc-9da3-d28ac08c3727-agent-certs\") pod \"konnectivity-agent-dxpvh\" (UID: \"a0f63ea2-446e-4fdc-9da3-d28ac08c3727\") " pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:25.904760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-device-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239283ff-d4c3-469a-b910-44d112a3c32a-tmp-dir\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-kubelet\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-ovnkube-config\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-hostroot\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-multus-certs\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-system-cni-dir\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsthv\" (UniqueName: \"kubernetes.io/projected/ddbffa5c-704d-4b71-a4b3-a19302668184-kube-api-access-jsthv\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239283ff-d4c3-469a-b910-44d112a3c32a-hosts-file\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdxh\" (UniqueName: \"kubernetes.io/projected/239283ff-d4c3-469a-b910-44d112a3c32a-kube-api-access-vrdxh\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-cnibin\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-conf-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-daemon-config\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-host\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-host-slash\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-socket-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-registration-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.905614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-etc-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77a32e00-3aac-4894-97c2-42327fc00c08-ovn-node-metrics-cert\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-cni-bin\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-cni-multus\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-cnibin\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.904982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysconfig\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-sys\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/222b1b29-9ac9-4418-b6c2-271ff86e4850-kubelet-config\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-env-overrides\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-socket-dir-parent\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-modprobe-d\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3499a298-490e-489b-a5fb-307cd8fcf6e9-host\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3499a298-490e-489b-a5fb-307cd8fcf6e9-serviceca\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.906523 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-node-log\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/222b1b29-9ac9-4418-b6c2-271ff86e4850-dbus\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905780 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-netns\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.905852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-etc-kubernetes\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-systemd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-var-lib-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-run-ovn\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:25.906541 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:25.906612 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret podName:222b1b29-9ac9-4418-b6c2-271ff86e4850 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:26.406581887 +0000 UTC m=+2.942488878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret") pod "global-pull-secret-syncer-d6f7p" (UID: "222b1b29-9ac9-4418-b6c2-271ff86e4850") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-run-netns\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-etc-openvswitch\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-kubelet\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/222b1b29-9ac9-4418-b6c2-271ff86e4850-kubelet-config\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-socket-dir-parent\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-conf-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.906989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-cni-multus\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.907350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907002 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-cnibin\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-k8s-cni-cncf-io\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-log-socket\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907426 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-cni-binary-copy\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e09db19d-ecec-41b5-85e3-164979161e08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e09db19d-ecec-41b5-85e3-164979161e08-system-cni-dir\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-cni-bin\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-systemd-units\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907716 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-env-overrides\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-cni-bin\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-run-multus-certs\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-slash\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-host-var-lib-kubelet\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908559 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.907647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-daemon-config\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908559 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-ovnkube-script-lib\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908559 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-hostroot\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908559 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-cni-netd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908559 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-system-cni-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908820 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-cnibin\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908820 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a32e00-3aac-4894-97c2-42327fc00c08-host-run-ovn-kubernetes\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908820 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-multus-cni-dir\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.908820 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77a32e00-3aac-4894-97c2-42327fc00c08-ovnkube-config\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.908820 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.908747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-cni-binary-copy\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:25.912001 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.911981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77a32e00-3aac-4894-97c2-42327fc00c08-ovn-node-metrics-cert\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.936403 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.936381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrrd\" (UniqueName: \"kubernetes.io/projected/77a32e00-3aac-4894-97c2-42327fc00c08-kube-api-access-nvrrd\") pod \"ovnkube-node-zcd47\" (UID: \"77a32e00-3aac-4894-97c2-42327fc00c08\") " pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:25.940098 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.940065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtnp\" (UniqueName: \"kubernetes.io/projected/e09db19d-ecec-41b5-85e3-164979161e08-kube-api-access-9rtnp\") pod \"multus-additional-cni-plugins-8kkp9\" (UID: \"e09db19d-ecec-41b5-85e3-164979161e08\") " pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:25.940402 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:25.940379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qhk\" (UniqueName: \"kubernetes.io/projected/469543e0-034f-49a3-ab0f-9f2ddeaf8fb8-kube-api-access-w9qhk\") pod \"multus-7j8x7\" (UID: \"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8\") " pod="openshift-multus/multus-7j8x7" Sep 11 11:30:26.006328 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsthv\" (UniqueName: \"kubernetes.io/projected/ddbffa5c-704d-4b71-a4b3-a19302668184-kube-api-access-jsthv\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.006328 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239283ff-d4c3-469a-b910-44d112a3c32a-hosts-file\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:26.006328 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdxh\" (UniqueName: \"kubernetes.io/projected/239283ff-d4c3-469a-b910-44d112a3c32a-kube-api-access-vrdxh\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239283ff-d4c3-469a-b910-44d112a3c32a-hosts-file\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-host\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-host-slash\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-socket-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-registration-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-host\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysconfig\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-host-slash\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:26.006569 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-sys\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-sys\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-modprobe-d\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysconfig\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006656 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-socket-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3499a298-490e-489b-a5fb-307cd8fcf6e9-host\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-registration-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.007106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006732 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-modprobe-d\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3499a298-490e-489b-a5fb-307cd8fcf6e9-host\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.006696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3499a298-490e-489b-a5fb-307cd8fcf6e9-serviceca\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.007764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4brc\" (UniqueName: \"kubernetes.io/projected/7eafd60c-bb2d-4473-ab8a-76180f836548-kube-api-access-n4brc\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.007811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-etc-selinux\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.007849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-tmp\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.007884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.007913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvm7c\" (UniqueName: \"kubernetes.io/projected/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-kube-api-access-pvm7c\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.007917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3499a298-490e-489b-a5fb-307cd8fcf6e9-serviceca\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008054 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-etc-selinux\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.008299 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.008170 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-iptables-alerter-script\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-systemd\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-var-lib-kubelet\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.008479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs podName:7eafd60c-bb2d-4473-ab8a-76180f836548 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:26.508456677 +0000 UTC m=+3.044363655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs") pod "network-metrics-daemon-59gwj" (UID: "7eafd60c-bb2d-4473-ab8a-76180f836548") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncg7d\" (UniqueName: \"kubernetes.io/projected/3499a298-490e-489b-a5fb-307cd8fcf6e9-kube-api-access-ncg7d\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-var-lib-kubelet\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-lib-modules\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysctl-d\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-run\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-tuned\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0f63ea2-446e-4fdc-9da3-d28ac08c3727-konnectivity-ca\") pod \"konnectivity-agent-dxpvh\" (UID: \"a0f63ea2-446e-4fdc-9da3-d28ac08c3727\") " pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpb9\" (UniqueName: \"kubernetes.io/projected/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-kube-api-access-8tpb9\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-kubernetes\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysctl-conf\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.008859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-sys-fs\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0f63ea2-446e-4fdc-9da3-d28ac08c3727-agent-certs\") pod \"konnectivity-agent-dxpvh\" (UID: \"a0f63ea2-446e-4fdc-9da3-d28ac08c3727\") " pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-iptables-alerter-script\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-device-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-device-dir\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.008990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239283ff-d4c3-469a-b910-44d112a3c32a-tmp-dir\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.009063 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-systemd\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.009116 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysctl-d\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.009184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-kubernetes\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.009307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-sysctl-conf\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.009311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239283ff-d4c3-469a-b910-44d112a3c32a-tmp-dir\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:26.009637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.009483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-lib-modules\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.010181 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.009951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0f63ea2-446e-4fdc-9da3-d28ac08c3727-konnectivity-ca\") pod \"konnectivity-agent-dxpvh\" (UID: \"a0f63ea2-446e-4fdc-9da3-d28ac08c3727\") " pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:26.010181 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.010037 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-run\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.010181 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.010102 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ddbffa5c-704d-4b71-a4b3-a19302668184-sys-fs\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.013425 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.013398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-etc-tuned\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.014952 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.014929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0f63ea2-446e-4fdc-9da3-d28ac08c3727-agent-certs\") pod \"konnectivity-agent-dxpvh\" (UID: \"a0f63ea2-446e-4fdc-9da3-d28ac08c3727\") " pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:26.015200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.015182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-tmp\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.024939 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.024915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsthv\" (UniqueName: \"kubernetes.io/projected/ddbffa5c-704d-4b71-a4b3-a19302668184-kube-api-access-jsthv\") pod \"aws-ebs-csi-driver-node-6tc84\" (UID: \"ddbffa5c-704d-4b71-a4b3-a19302668184\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.026398 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.026364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdxh\" (UniqueName: \"kubernetes.io/projected/239283ff-d4c3-469a-b910-44d112a3c32a-kube-api-access-vrdxh\") pod \"node-resolver-fzqcl\" (UID: \"239283ff-d4c3-469a-b910-44d112a3c32a\") " pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:26.027836 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.027816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4brc\" (UniqueName: \"kubernetes.io/projected/7eafd60c-bb2d-4473-ab8a-76180f836548-kube-api-access-n4brc\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:26.031242 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.031216 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpb9\" (UniqueName: \"kubernetes.io/projected/5b67833d-ab3e-4a80-b687-3ab730fd8a3e-kube-api-access-8tpb9\") pod \"tuned-k5mxn\" (UID: \"5b67833d-ab3e-4a80-b687-3ab730fd8a3e\") " pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.031716 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.031696 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:30:26.031889 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.031720 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:30:26.031889 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.031752 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jldh6 for pod openshift-network-diagnostics/network-check-target-prvf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:26.031889 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.031812 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6 podName:ba4cf6d4-6127-4838-8e66-f71984a95365 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:26.531793422 +0000 UTC m=+3.067700413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jldh6" (UniqueName: "kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6") pod "network-check-target-prvf7" (UID: "ba4cf6d4-6127-4838-8e66-f71984a95365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:26.034627 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.034584 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvm7c\" (UniqueName: \"kubernetes.io/projected/7d82b26a-e9ab-4593-9179-fa7a73dd9cf6-kube-api-access-pvm7c\") pod \"iptables-alerter-lp96t\" (UID: \"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6\") " pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:26.035605 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.035544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncg7d\" (UniqueName: \"kubernetes.io/projected/3499a298-490e-489b-a5fb-307cd8fcf6e9-kube-api-access-ncg7d\") pod \"node-ca-4s48h\" (UID: \"3499a298-490e-489b-a5fb-307cd8fcf6e9\") " pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:26.097577 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.097547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:26.105633 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.105613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7j8x7" Sep 11 11:30:26.116026 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.115999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" Sep 11 11:30:26.123927 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.123905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lp96t" Sep 11 11:30:26.131742 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.131722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" Sep 11 11:30:26.141755 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.141737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:26.147591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.147568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" Sep 11 11:30:26.157395 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.157375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fzqcl" Sep 11 11:30:26.162284 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.162265 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4s48h" Sep 11 11:30:26.212048 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.212021 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:30:26.412155 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.412054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:26.412316 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.412252 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:26.412316 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.412309 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret podName:222b1b29-9ac9-4418-b6c2-271ff86e4850 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:27.412294245 +0000 UTC m=+3.948201222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret") pod "global-pull-secret-syncer-d6f7p" (UID: "222b1b29-9ac9-4418-b6c2-271ff86e4850") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:26.512645 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.512611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:26.512839 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.512809 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:26.512915 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.512899 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs podName:7eafd60c-bb2d-4473-ab8a-76180f836548 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:27.512875751 +0000 UTC m=+4.048782770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs") pod "network-metrics-daemon-59gwj" (UID: "7eafd60c-bb2d-4473-ab8a-76180f836548") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:26.613228 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.613186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:26.613385 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.613359 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:30:26.613385 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.613385 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:30:26.613460 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.613395 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jldh6 for pod openshift-network-diagnostics/network-check-target-prvf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:26.613460 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.613454 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6 podName:ba4cf6d4-6127-4838-8e66-f71984a95365 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:27.613438698 +0000 UTC m=+4.149345682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jldh6" (UniqueName: "kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6") pod "network-check-target-prvf7" (UID: "ba4cf6d4-6127-4838-8e66-f71984a95365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:26.843052 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.842954 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 11:25:26 +0000 UTC" deadline="2028-06-12 18:50:15.045080396 +0000 UTC" Sep 11 11:30:26.843052 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.842990 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15367h19m48.202093753s" Sep 11 11:30:26.894700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.894668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:26.894875 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:26.894668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:26.894875 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.894843 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:26.894994 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:26.894967 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:26.990244 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:26.990050 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a32e00_3aac_4894_97c2_42327fc00c08.slice/crio-4c6316e87b006ecefdffc09924646e7654a219b56ee505185feb9a608a385518 WatchSource:0}: Error finding container 4c6316e87b006ecefdffc09924646e7654a219b56ee505185feb9a608a385518: Status 404 returned error can't find the container with id 4c6316e87b006ecefdffc09924646e7654a219b56ee505185feb9a608a385518 Sep 11 11:30:26.991817 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:26.991798 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3499a298_490e_489b_a5fb_307cd8fcf6e9.slice/crio-54d80f502ddffd4d57d35c0643cf9760f37d8adfb1ade3e90d1bfe9578be73ef WatchSource:0}: Error finding container 54d80f502ddffd4d57d35c0643cf9760f37d8adfb1ade3e90d1bfe9578be73ef: Status 404 returned error can't find the container with id 54d80f502ddffd4d57d35c0643cf9760f37d8adfb1ade3e90d1bfe9578be73ef Sep 11 11:30:26.993170 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:26.993093 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82b26a_e9ab_4593_9179_fa7a73dd9cf6.slice/crio-b1ea3d2a7555b717c36f8db48b7ee8e7ef652c8f015a2cb55b1d179fe7379da7 WatchSource:0}: Error finding container b1ea3d2a7555b717c36f8db48b7ee8e7ef652c8f015a2cb55b1d179fe7379da7: Status 404 returned error can't find the container with id b1ea3d2a7555b717c36f8db48b7ee8e7ef652c8f015a2cb55b1d179fe7379da7 Sep 11 11:30:26.994747 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:26.994136 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b67833d_ab3e_4a80_b687_3ab730fd8a3e.slice/crio-e0b09e6d31b57ea8632820cd52e2bf15164b2a785d5cc83a5de55540e74c7a1c WatchSource:0}: Error finding container e0b09e6d31b57ea8632820cd52e2bf15164b2a785d5cc83a5de55540e74c7a1c: Status 404 returned error can't find the container with id e0b09e6d31b57ea8632820cd52e2bf15164b2a785d5cc83a5de55540e74c7a1c Sep 11 11:30:26.997921 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:26.997881 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239283ff_d4c3_469a_b910_44d112a3c32a.slice/crio-6d07ec37c2b7d6f4cc02b6a4f8ab04caf0129b30efb962a0e02831eb7d2eee3a WatchSource:0}: Error finding container 6d07ec37c2b7d6f4cc02b6a4f8ab04caf0129b30efb962a0e02831eb7d2eee3a: Status 404 returned error can't find the container with id 6d07ec37c2b7d6f4cc02b6a4f8ab04caf0129b30efb962a0e02831eb7d2eee3a Sep 11 11:30:26.998819 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:26.998774 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbffa5c_704d_4b71_a4b3_a19302668184.slice/crio-3e1b55e8868eb5880e7086294368b994ced333d6f16731784e1793f57ed2de2d WatchSource:0}: Error finding container 3e1b55e8868eb5880e7086294368b994ced333d6f16731784e1793f57ed2de2d: Status 404 returned error can't find the container with id 3e1b55e8868eb5880e7086294368b994ced333d6f16731784e1793f57ed2de2d Sep 11 11:30:26.999945 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:26.999812 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469543e0_034f_49a3_ab0f_9f2ddeaf8fb8.slice/crio-cf476649b3c0e5eb060ffc85d3e935303d3cbd0d4845c82067f283396957fe30 WatchSource:0}: Error finding container cf476649b3c0e5eb060ffc85d3e935303d3cbd0d4845c82067f283396957fe30: Status 404 returned error can't find the container with id cf476649b3c0e5eb060ffc85d3e935303d3cbd0d4845c82067f283396957fe30 Sep 11 11:30:27.000784 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:27.000748 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09db19d_ecec_41b5_85e3_164979161e08.slice/crio-5bae0a904250ecdcb60805ddd5d3d0424e7f70109520145bca22ab5db57aa9ef WatchSource:0}: Error finding container 5bae0a904250ecdcb60805ddd5d3d0424e7f70109520145bca22ab5db57aa9ef: Status 404 returned error can't find the container with id 5bae0a904250ecdcb60805ddd5d3d0424e7f70109520145bca22ab5db57aa9ef Sep 11 11:30:27.003201 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:27.002328 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f63ea2_446e_4fdc_9da3_d28ac08c3727.slice/crio-7d8635f67982c9aae76e806b942dc03d5af4973d2d6fa57e0a4cfc7e40bd8d80 WatchSource:0}: Error finding container 7d8635f67982c9aae76e806b942dc03d5af4973d2d6fa57e0a4cfc7e40bd8d80: Status 404 returned error can't find the container with id 7d8635f67982c9aae76e806b942dc03d5af4973d2d6fa57e0a4cfc7e40bd8d80 Sep 11 11:30:27.420956 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.420922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:27.421077 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.421059 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:27.421117 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.421112 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret podName:222b1b29-9ac9-4418-b6c2-271ff86e4850 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:29.421097511 +0000 UTC m=+5.957004488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret") pod "global-pull-secret-syncer-d6f7p" (UID: "222b1b29-9ac9-4418-b6c2-271ff86e4850") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:27.522038 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.521924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:27.522222 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.522078 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:27.522222 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.522141 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs podName:7eafd60c-bb2d-4473-ab8a-76180f836548 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:29.522122327 +0000 UTC m=+6.058029310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs") pod "network-metrics-daemon-59gwj" (UID: "7eafd60c-bb2d-4473-ab8a-76180f836548") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:27.623391 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.622704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:27.623391 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.622923 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:30:27.623391 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.622946 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:30:27.623391 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.622960 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jldh6 for pod openshift-network-diagnostics/network-check-target-prvf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:27.623391 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.623023 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6 podName:ba4cf6d4-6127-4838-8e66-f71984a95365 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:29.623002636 +0000 UTC m=+6.158909632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jldh6" (UniqueName: "kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6") pod "network-check-target-prvf7" (UID: "ba4cf6d4-6127-4838-8e66-f71984a95365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:27.844044 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.843933 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 11:25:26 +0000 UTC" deadline="2028-06-19 04:03:35.751163874 +0000 UTC" Sep 11 11:30:27.844044 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.843980 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15520h33m7.907188581s" Sep 11 11:30:27.902980 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.902945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:27.903201 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:27.903139 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:27.923441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.923398 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" event={"ID":"ddbffa5c-704d-4b71-a4b3-a19302668184","Type":"ContainerStarted","Data":"3e1b55e8868eb5880e7086294368b994ced333d6f16731784e1793f57ed2de2d"} Sep 11 11:30:27.924095 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.924014 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fzqcl" event={"ID":"239283ff-d4c3-469a-b910-44d112a3c32a","Type":"ContainerStarted","Data":"6d07ec37c2b7d6f4cc02b6a4f8ab04caf0129b30efb962a0e02831eb7d2eee3a"} Sep 11 11:30:27.937184 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.937150 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" event={"ID":"5b67833d-ab3e-4a80-b687-3ab730fd8a3e","Type":"ContainerStarted","Data":"e0b09e6d31b57ea8632820cd52e2bf15164b2a785d5cc83a5de55540e74c7a1c"} Sep 11 11:30:27.956550 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.956517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"4c6316e87b006ecefdffc09924646e7654a219b56ee505185feb9a608a385518"} Sep 11 11:30:27.975818 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.975786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dxpvh" event={"ID":"a0f63ea2-446e-4fdc-9da3-d28ac08c3727","Type":"ContainerStarted","Data":"7d8635f67982c9aae76e806b942dc03d5af4973d2d6fa57e0a4cfc7e40bd8d80"} Sep 11 11:30:27.981414 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.981383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7j8x7" event={"ID":"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8","Type":"ContainerStarted","Data":"cf476649b3c0e5eb060ffc85d3e935303d3cbd0d4845c82067f283396957fe30"} Sep 11 11:30:27.988887 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.988854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lp96t" event={"ID":"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6","Type":"ContainerStarted","Data":"b1ea3d2a7555b717c36f8db48b7ee8e7ef652c8f015a2cb55b1d179fe7379da7"} Sep 11 11:30:27.991431 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.991377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4s48h" event={"ID":"3499a298-490e-489b-a5fb-307cd8fcf6e9","Type":"ContainerStarted","Data":"54d80f502ddffd4d57d35c0643cf9760f37d8adfb1ade3e90d1bfe9578be73ef"} Sep 11 11:30:27.996613 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:27.996560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" event={"ID":"c14bf53574a5faf7f92e569f4d21b76b","Type":"ContainerStarted","Data":"95a3eca9691c7d2fd1f19137984f473cf4c8aceea2b99d6edff2fe73d4ebc226"} Sep 11 11:30:28.001567 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:28.001530 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4132a9a58a5c237803951588c311249" containerID="bd4ba5898f2f5327e35004c40e86249600e42294d45d0337d39a4ea8368c885b" exitCode=0 Sep 11 11:30:28.001681 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:28.001617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" event={"ID":"e4132a9a58a5c237803951588c311249","Type":"ContainerDied","Data":"bd4ba5898f2f5327e35004c40e86249600e42294d45d0337d39a4ea8368c885b"} Sep 11 11:30:28.010239 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:28.010203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerStarted","Data":"5bae0a904250ecdcb60805ddd5d3d0424e7f70109520145bca22ab5db57aa9ef"} Sep 11 11:30:28.032537 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:28.032473 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-17.ec2.internal" podStartSLOduration=3.032453516 podStartE2EDuration="3.032453516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:28.013773488 +0000 UTC m=+4.549680491" watchObservedRunningTime="2026-09-11 11:30:28.032453516 +0000 UTC m=+4.568360519" Sep 11 11:30:28.894514 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:28.894474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:28.895194 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:28.894610 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:28.895194 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:28.895067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:28.895308 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:28.895189 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:29.026896 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:29.026859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" event={"ID":"e4132a9a58a5c237803951588c311249","Type":"ContainerStarted","Data":"b6f73e6e493ab307531f8b1c28644cbca20248ae7369644be36a588f245b627a"} Sep 11 11:30:29.439733 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:29.439092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:29.439733 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.439281 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:29.439733 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.439349 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret podName:222b1b29-9ac9-4418-b6c2-271ff86e4850 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:33.439329501 +0000 UTC m=+9.975236481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret") pod "global-pull-secret-syncer-d6f7p" (UID: "222b1b29-9ac9-4418-b6c2-271ff86e4850") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:29.540383 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:29.540343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:29.540579 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.540524 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:29.540640 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.540587 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs podName:7eafd60c-bb2d-4473-ab8a-76180f836548 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:33.540568726 +0000 UTC m=+10.076475717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs") pod "network-metrics-daemon-59gwj" (UID: "7eafd60c-bb2d-4473-ab8a-76180f836548") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:29.641642 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:29.641539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:29.641895 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.641780 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:30:29.641895 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.641814 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:30:29.641895 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.641829 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jldh6 for pod openshift-network-diagnostics/network-check-target-prvf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:29.642111 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.641899 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6 podName:ba4cf6d4-6127-4838-8e66-f71984a95365 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:33.641869633 +0000 UTC m=+10.177776625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jldh6" (UniqueName: "kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6") pod "network-check-target-prvf7" (UID: "ba4cf6d4-6127-4838-8e66-f71984a95365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:29.896483 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:29.896395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:29.896992 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:29.896539 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:30.894108 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:30.894074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:30.894344 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:30.894268 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:30.894759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:30.894710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:30.894861 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:30.894831 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:31.902809 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:31.899630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:31.902809 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:31.899826 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:32.894103 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:32.894024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:32.894266 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:32.894160 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:32.894266 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:32.894208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:32.894365 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:32.894331 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:33.472100 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:33.472052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:33.472566 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.472172 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:33.472566 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.472237 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret podName:222b1b29-9ac9-4418-b6c2-271ff86e4850 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:41.472218183 +0000 UTC m=+18.008125167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret") pod "global-pull-secret-syncer-d6f7p" (UID: "222b1b29-9ac9-4418-b6c2-271ff86e4850") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:33.572956 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:33.572928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:33.573154 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.573040 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:33.573154 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.573101 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs podName:7eafd60c-bb2d-4473-ab8a-76180f836548 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:41.573086252 +0000 UTC m=+18.108993227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs") pod "network-metrics-daemon-59gwj" (UID: "7eafd60c-bb2d-4473-ab8a-76180f836548") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:33.674006 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:33.673879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:33.674182 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.674042 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:30:33.674182 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.674069 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:30:33.674182 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.674080 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jldh6 for pod openshift-network-diagnostics/network-check-target-prvf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:33.674182 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.674123 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6 podName:ba4cf6d4-6127-4838-8e66-f71984a95365 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:41.674110836 +0000 UTC m=+18.210017811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jldh6" (UniqueName: "kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6") pod "network-check-target-prvf7" (UID: "ba4cf6d4-6127-4838-8e66-f71984a95365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:33.895710 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:33.895278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:33.895710 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:33.895369 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:34.894761 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:34.894113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:34.894761 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:34.894243 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:34.894761 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:34.894608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:34.894761 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:34.894706 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:35.895900 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:35.895823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:35.896353 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:35.895945 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:36.894262 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:36.894232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:36.894262 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:36.894262 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:36.894461 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:36.894336 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:36.894461 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:36.894392 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:37.894292 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:37.894262 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:37.894771 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:37.894357 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:38.894832 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:38.894591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:38.895295 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:38.894663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:38.895295 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:38.894908 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:38.895295 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:38.895028 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:39.894536 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:39.894498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:39.894704 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:39.894648 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:40.894114 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:40.894076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:40.894622 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:40.894234 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:40.894622 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:40.894613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:40.894743 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:40.894696 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:41.530849 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:41.530807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:41.531024 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.530962 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:41.531086 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.531038 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret podName:222b1b29-9ac9-4418-b6c2-271ff86e4850 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:57.53101659 +0000 UTC m=+34.066923567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret") pod "global-pull-secret-syncer-d6f7p" (UID: "222b1b29-9ac9-4418-b6c2-271ff86e4850") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:41.631919 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:41.631887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:41.632080 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.632043 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:41.632154 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.632118 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs podName:7eafd60c-bb2d-4473-ab8a-76180f836548 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:57.632095718 +0000 UTC m=+34.168002716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs") pod "network-metrics-daemon-59gwj" (UID: "7eafd60c-bb2d-4473-ab8a-76180f836548") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:41.733199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:41.733171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:41.733325 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.733299 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:30:41.733325 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.733316 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:30:41.733325 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.733326 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jldh6 for pod openshift-network-diagnostics/network-check-target-prvf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:41.733428 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.733368 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6 podName:ba4cf6d4-6127-4838-8e66-f71984a95365 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:57.733355259 +0000 UTC m=+34.269262234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jldh6" (UniqueName: "kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6") pod "network-check-target-prvf7" (UID: "ba4cf6d4-6127-4838-8e66-f71984a95365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:41.894641 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:41.894621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:41.894983 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:41.894727 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:42.894669 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:42.894638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:42.895115 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:42.894647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:42.895115 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:42.894786 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:42.895115 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:42.894836 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:43.895542 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:43.895512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:43.895957 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:43.895606 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:44.894900 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:44.894865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:44.895058 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:44.894979 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:44.895058 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:44.895039 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:44.895188 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:44.895147 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:45.894693 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:45.894526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:45.895329 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:45.894796 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:46.065873 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.065789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"36fed63ac7298636b5c6d37823dc5fde64c1fe9868afa3f7490422f4a44159d3"} Sep 11 11:30:46.065873 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.065867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"9fa5b71304de8ea9962ec28fc552da186eafd8465bc5f6b35c88d53c4a6c6855"} Sep 11 11:30:46.066074 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.065915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"182aa50642e3ebfd351e02d63dd5de0d0a909a8cebecd2677c94a4c566587243"} Sep 11 11:30:46.066074 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.065964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"df40abeadd817f787f43ff3a6d08976ddaf123461ab2f2516a30f089177a8fde"} Sep 11 11:30:46.066074 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.066009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"368ddc68bece6fcbaf59152b4fb4eb211b66f6cd1a544c0a2dbe538a3509a743"} Sep 11 11:30:46.066074 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.066041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"2fb2eccf4af0011fdd80e54cac6f681252b20c8ff03fe8af349dcfc503b5a7d9"} Sep 11 11:30:46.067589 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.067556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dxpvh" event={"ID":"a0f63ea2-446e-4fdc-9da3-d28ac08c3727","Type":"ContainerStarted","Data":"e421078e8fd9bf906b1cad8801d2291629a5c97c3e2a2444185a4913a8af71d9"} Sep 11 11:30:46.069311 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.069283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7j8x7" event={"ID":"469543e0-034f-49a3-ab0f-9f2ddeaf8fb8","Type":"ContainerStarted","Data":"c08e4ad148052c490e3ab5dbb6b8dd9ceaf0039f7467c9a9dab5509e24a74d09"} Sep 11 11:30:46.070907 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.070885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4s48h" event={"ID":"3499a298-490e-489b-a5fb-307cd8fcf6e9","Type":"ContainerStarted","Data":"be7df2eb1a082046f3dc2beacf907242cf903e1b15f8b432b546b2c69e52ef4a"} Sep 11 11:30:46.072284 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.072265 2579 generic.go:356] "Generic (PLEG): container finished" podID="e09db19d-ecec-41b5-85e3-164979161e08" containerID="916f3972cdcd7caab8d832cedd34ec31c443764b7c6df6db13e8ab46145ab061" exitCode=0 Sep 11 11:30:46.072368 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.072328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerDied","Data":"916f3972cdcd7caab8d832cedd34ec31c443764b7c6df6db13e8ab46145ab061"} Sep 11 11:30:46.074057 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.074034 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" event={"ID":"ddbffa5c-704d-4b71-a4b3-a19302668184","Type":"ContainerStarted","Data":"d987280735e83612e50109e2372f75cdd9d5175e4584b5dae0e20ebd2f63824a"} Sep 11 11:30:46.075829 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.075811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fzqcl" event={"ID":"239283ff-d4c3-469a-b910-44d112a3c32a","Type":"ContainerStarted","Data":"376732dae2d787d298e514bcc5dc0bac87d16d44237365358f244e8a51eccc1d"} Sep 11 11:30:46.077417 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.077394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" event={"ID":"5b67833d-ab3e-4a80-b687-3ab730fd8a3e","Type":"ContainerStarted","Data":"b1eef3eb7b41425e7c1582a8d758958cd5a1009fc1ae1060dc691714ea370302"} Sep 11 11:30:46.083692 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.083656 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dxpvh" podStartSLOduration=7.35107778 podStartE2EDuration="22.083644691s" totalImagesPullingTime="14.732566911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:46.083436305 +0000 UTC m=+22.619343302" watchObservedRunningTime="2026-09-11 11:30:46.083644691 +0000 UTC m=+22.619551688" Sep 11 11:30:46.083802 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.083760 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-17.ec2.internal" podStartSLOduration=21.083755057 podStartE2EDuration="21.083755057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:29.045062834 +0000 UTC m=+5.580969837" watchObservedRunningTime="2026-09-11 11:30:46.083755057 +0000 UTC m=+22.619662053" Sep 11 11:30:46.115133 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.115095 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k5mxn" podStartSLOduration=3.842442722 podStartE2EDuration="22.115082418s" totalImagesPullingTime="18.272639696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:46.100474225 +0000 UTC m=+22.636381222" watchObservedRunningTime="2026-09-11 11:30:46.115082418 +0000 UTC m=+22.650989417" Sep 11 11:30:46.115250 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.115218 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fzqcl" podStartSLOduration=3.862901223 podStartE2EDuration="22.115209635s" totalImagesPullingTime="18.252308412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:46.114787166 +0000 UTC m=+22.650694165" watchObservedRunningTime="2026-09-11 11:30:46.115209635 +0000 UTC m=+22.651116635" Sep 11 11:30:46.128493 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.128460 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4s48h" podStartSLOduration=3.878177807 podStartE2EDuration="22.128448759s" totalImagesPullingTime="18.250270952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:46.128364384 +0000 UTC m=+22.664271381" watchObservedRunningTime="2026-09-11 11:30:46.128448759 +0000 UTC m=+22.664355756" Sep 11 11:30:46.146016 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.145980 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7j8x7" podStartSLOduration=3.8377768960000003 podStartE2EDuration="22.145968107s" totalImagesPullingTime="18.308191211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:46.145338248 +0000 UTC m=+22.681245248" watchObservedRunningTime="2026-09-11 11:30:46.145968107 +0000 UTC m=+22.681875106" Sep 11 11:30:46.780366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.780342 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 11:30:46.877999 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.877894 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T11:30:46.780362352Z","UUID":"0bbccd2d-0f66-4acf-ace4-59982859495e","Handler":null,"Name":"","Endpoint":""} Sep 11 11:30:46.879195 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.879177 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 11:30:46.879269 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.879206 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 11:30:46.894245 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.894224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:46.894330 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:46.894258 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:46.894365 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:46.894325 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:46.894464 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:46.894444 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:47.084843 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:47.084753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lp96t" event={"ID":"7d82b26a-e9ab-4593-9179-fa7a73dd9cf6","Type":"ContainerStarted","Data":"cafc5829a309380be67dbef538ba7fb93455a5cbc8371a64dcc97b132935ca7f"} Sep 11 11:30:47.087195 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:47.087129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" event={"ID":"ddbffa5c-704d-4b71-a4b3-a19302668184","Type":"ContainerStarted","Data":"2f9d81252246d982f69a58e95fda66ceb9ca320af91d09206438cd962a52044f"} Sep 11 11:30:47.100871 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:47.100824 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lp96t" podStartSLOduration=4.840231648 podStartE2EDuration="23.100807208s" totalImagesPullingTime="18.26057556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:47.100314584 +0000 UTC m=+23.636221581" watchObservedRunningTime="2026-09-11 11:30:47.100807208 +0000 UTC m=+23.636714205" Sep 11 11:30:47.894929 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:47.894756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:47.895041 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:47.895022 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:48.092241 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:48.092149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" event={"ID":"ddbffa5c-704d-4b71-a4b3-a19302668184","Type":"ContainerStarted","Data":"2dd734da2beb2ef0348ff1a58cd4962cdce0b6fc09ecf3c60fe1b931812e453b"} Sep 11 11:30:48.096036 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:48.096003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"ab1215fff45723d83f9fd55508429b175930ff86aa3122ec87aa078c9f91769e"} Sep 11 11:30:48.110037 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:48.109991 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tc84" podStartSLOduration=3.708973856 podStartE2EDuration="24.10997542s" totalImagesPullingTime="20.401001564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:48.109851157 +0000 UTC m=+24.645758167" watchObservedRunningTime="2026-09-11 11:30:48.10997542 +0000 UTC m=+24.645882419" Sep 11 11:30:48.894174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:48.894142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:48.894174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:48.894182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:48.894431 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:48.894273 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:48.894489 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:48.894443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:49.894112 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:49.894079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:49.894474 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:49.894214 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:50.256242 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:50.256152 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:50.256908 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:50.256884 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:50.895084 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:50.894928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:50.895363 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:50.894934 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:50.895363 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:50.895150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:50.895363 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:50.895231 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:51.104521 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:51.104499 2579 generic.go:356] "Generic (PLEG): container finished" podID="e09db19d-ecec-41b5-85e3-164979161e08" containerID="dcc183858d21fa6904d3b4712af337f613c1f044aa3e9424a7d330286ea9ff02" exitCode=0 Sep 11 11:30:51.104658 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:51.104589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerDied","Data":"dcc183858d21fa6904d3b4712af337f613c1f044aa3e9424a7d330286ea9ff02"} Sep 11 11:30:51.108517 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:51.108491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" event={"ID":"77a32e00-3aac-4894-97c2-42327fc00c08","Type":"ContainerStarted","Data":"6204d3b04c98801266e684c0b3c6f519145bb59b3fa5e185b74a317b0c40e68d"} Sep 11 11:30:51.108839 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:51.108819 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:51.109184 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:51.109166 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dxpvh" Sep 11 11:30:51.167191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:51.167155 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" podStartSLOduration=8.84120521 podStartE2EDuration="27.167142773s" totalImagesPullingTime="18.325937563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:30:51.166290049 +0000 UTC m=+27.702197057" watchObservedRunningTime="2026-09-11 11:30:51.167142773 +0000 UTC m=+27.703049770" Sep 11 11:30:51.894955 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:51.894925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:51.895114 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:51.895021 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:52.112880 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.112854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:52.113005 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.112924 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:52.113005 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.112954 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:52.130711 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.130684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:52.130851 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.130821 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:30:52.346941 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.346912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-prvf7"] Sep 11 11:30:52.347094 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.347060 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:52.347188 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:52.347160 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:52.347540 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.347515 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6f7p"] Sep 11 11:30:52.347635 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.347617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:52.347809 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:52.347785 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:52.349922 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.349900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-59gwj"] Sep 11 11:30:52.350020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:52.349998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:52.350115 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:52.350096 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:53.116458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:53.116429 2579 generic.go:356] "Generic (PLEG): container finished" podID="e09db19d-ecec-41b5-85e3-164979161e08" containerID="897aa7dd4a113d20691d578ac24587b9c7d0ff6ade7ac578a87920b95681d07a" exitCode=0 Sep 11 11:30:53.117017 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:53.116514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerDied","Data":"897aa7dd4a113d20691d578ac24587b9c7d0ff6ade7ac578a87920b95681d07a"} Sep 11 11:30:53.895568 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:53.895342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:53.895661 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:53.895401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:53.895746 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:53.895672 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:53.895746 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:53.895417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:53.895852 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:53.895758 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:53.895852 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:53.895801 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:54.122249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:54.121821 2579 generic.go:356] "Generic (PLEG): container finished" podID="e09db19d-ecec-41b5-85e3-164979161e08" containerID="22a6459407f8fb34caade6fc0deefbd4edf30fa075ef9512fed4a1540c6aa495" exitCode=0 Sep 11 11:30:54.122249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:54.121924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerDied","Data":"22a6459407f8fb34caade6fc0deefbd4edf30fa075ef9512fed4a1540c6aa495"} Sep 11 11:30:55.894965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:55.894932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:55.895413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:55.894932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:55.895413 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:55.895055 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:55.895413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:55.894933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:55.895413 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:55.895181 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:55.895413 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:55.895263 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:57.551387 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:57.551346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:57.551915 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.551513 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:57.551915 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.551579 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret podName:222b1b29-9ac9-4418-b6c2-271ff86e4850 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:29.551560881 +0000 UTC m=+66.087467856 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret") pod "global-pull-secret-syncer-d6f7p" (UID: "222b1b29-9ac9-4418-b6c2-271ff86e4850") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:30:57.652479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:57.652447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:57.652646 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.652564 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:57.652646 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.652623 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs podName:7eafd60c-bb2d-4473-ab8a-76180f836548 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:29.652604595 +0000 UTC m=+66.188511570 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs") pod "network-metrics-daemon-59gwj" (UID: "7eafd60c-bb2d-4473-ab8a-76180f836548") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:30:57.753378 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:57.753344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:57.753577 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.753501 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:30:57.753577 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.753521 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:30:57.753577 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.753532 2579 projected.go:196] Error preparing data for projected volume kube-api-access-jldh6 for pod openshift-network-diagnostics/network-check-target-prvf7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:57.753781 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.753600 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6 podName:ba4cf6d4-6127-4838-8e66-f71984a95365 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:29.753579823 +0000 UTC m=+66.289486811 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jldh6" (UniqueName: "kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6") pod "network-check-target-prvf7" (UID: "ba4cf6d4-6127-4838-8e66-f71984a95365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:30:57.894787 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:57.894754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:57.894966 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:57.894754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:57.894966 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.894882 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-prvf7" podUID="ba4cf6d4-6127-4838-8e66-f71984a95365" Sep 11 11:30:57.894966 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:57.894759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:57.895118 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.894986 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-59gwj" podUID="7eafd60c-bb2d-4473-ab8a-76180f836548" Sep 11 11:30:57.895118 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:57.895028 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6f7p" podUID="222b1b29-9ac9-4418-b6c2-271ff86e4850" Sep 11 11:30:59.135292 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.135260 2579 generic.go:356] "Generic (PLEG): container finished" podID="e09db19d-ecec-41b5-85e3-164979161e08" containerID="2363d96e9a3bfc63a5908a7a9e9b3d98771e25999ce0e78ff1b6337df374688f" exitCode=0 Sep 11 11:30:59.135613 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.135329 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerDied","Data":"2363d96e9a3bfc63a5908a7a9e9b3d98771e25999ce0e78ff1b6337df374688f"} Sep 11 11:30:59.200307 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.200280 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-17.ec2.internal" event="NodeReady" Sep 11 11:30:59.200441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.200429 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 11:30:59.240593 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.240569 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f5dcd6d57-m697t"] Sep 11 11:30:59.280638 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.280614 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-shx9t"] Sep 11 11:30:59.280784 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.280768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.283335 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.283311 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 11:30:59.283442 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.283311 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 11:30:59.283780 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.283759 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 11:30:59.284054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.283814 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mqz7m\"" Sep 11 11:30:59.294320 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.294298 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 11:30:59.308913 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.308889 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6h82f"] Sep 11 11:30:59.309021 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.309005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.311996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.311855 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-trsxb\"" Sep 11 11:30:59.311996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.311920 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 11:30:59.312156 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.312023 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 11:30:59.312210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.312180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 11:30:59.327127 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.327109 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f5dcd6d57-m697t"] Sep 11 11:30:59.327208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.327137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shx9t"] Sep 11 11:30:59.327208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.327151 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6h82f"] Sep 11 11:30:59.327273 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.327208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.330817 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.330791 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zhwh9\"" Sep 11 11:30:59.331165 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.331151 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 11:30:59.331259 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.331244 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 11:30:59.365980 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.365961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-image-registry-private-configuration\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.366089 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.365989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-trusted-ca\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.366089 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.366006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-installation-pull-secrets\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.366089 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.366029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-bound-sa-token\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.366208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.366113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.366208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.366157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckr7\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-kube-api-access-9ckr7\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.366208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.366182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-ca-trust-extracted\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.366208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.366199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-certificates\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.466533 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbc8c5a1-ddea-4819-909d-23aae4d3e4f7-cert\") pod \"ingress-canary-shx9t\" (UID: \"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7\") " pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.466533 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-ca-trust-extracted\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.466676 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33290017-2b75-42ad-be09-29ee5b7e746e-config-volume\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.466676 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9s22\" (UniqueName: \"kubernetes.io/projected/33290017-2b75-42ad-be09-29ee5b7e746e-kube-api-access-k9s22\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.466676 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-certificates\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.466676 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-image-registry-private-configuration\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.466908 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-trusted-ca\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.466908 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-installation-pull-secrets\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.466908 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-bound-sa-token\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.467028 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.466940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.467028 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.467006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckr7\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-kube-api-access-9ckr7\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.467126 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.467034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.467126 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.467066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/bbc8c5a1-ddea-4819-909d-23aae4d3e4f7-kube-api-access-tt9sc\") pod \"ingress-canary-shx9t\" (UID: \"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7\") " pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.467126 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.467087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33290017-2b75-42ad-be09-29ee5b7e746e-tmp-dir\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.467126 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.467106 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:30:59.467126 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.467123 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5dcd6d57-m697t: secret "image-registry-tls" not found Sep 11 11:30:59.467356 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.467174 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls podName:9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933 nodeName:}" failed. No retries permitted until 2026-09-11 11:30:59.967156234 +0000 UTC m=+36.503063210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls") pod "image-registry-6f5dcd6d57-m697t" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933") : secret "image-registry-tls" not found Sep 11 11:30:59.467356 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.467183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-ca-trust-extracted\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.467640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.467502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-certificates\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.467759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.467740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-trusted-ca\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.470637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.470617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-installation-pull-secrets\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.470637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.470630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-image-registry-private-configuration\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.477788 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.477766 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-bound-sa-token\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.477861 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.477805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckr7\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-kube-api-access-9ckr7\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.567941 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.567919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.568020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.567945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/bbc8c5a1-ddea-4819-909d-23aae4d3e4f7-kube-api-access-tt9sc\") pod \"ingress-canary-shx9t\" (UID: \"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7\") " pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.568020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.567959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33290017-2b75-42ad-be09-29ee5b7e746e-tmp-dir\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.568020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.567975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbc8c5a1-ddea-4819-909d-23aae4d3e4f7-cert\") pod \"ingress-canary-shx9t\" (UID: \"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7\") " pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.568020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.567990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33290017-2b75-42ad-be09-29ee5b7e746e-config-volume\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.568199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.568036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9s22\" (UniqueName: \"kubernetes.io/projected/33290017-2b75-42ad-be09-29ee5b7e746e-kube-api-access-k9s22\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.568199 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.568062 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:30:59.568199 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.568109 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls podName:33290017-2b75-42ad-be09-29ee5b7e746e nodeName:}" failed. No retries permitted until 2026-09-11 11:31:00.068096189 +0000 UTC m=+36.604003167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls") pod "dns-default-6h82f" (UID: "33290017-2b75-42ad-be09-29ee5b7e746e") : secret "dns-default-metrics-tls" not found Sep 11 11:30:59.568345 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.568278 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33290017-2b75-42ad-be09-29ee5b7e746e-tmp-dir\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.568614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.568598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/33290017-2b75-42ad-be09-29ee5b7e746e-config-volume\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.570083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.570062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbc8c5a1-ddea-4819-909d-23aae4d3e4f7-cert\") pod \"ingress-canary-shx9t\" (UID: \"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7\") " pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.577311 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.577290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9sc\" (UniqueName: \"kubernetes.io/projected/bbc8c5a1-ddea-4819-909d-23aae4d3e4f7-kube-api-access-tt9sc\") pod \"ingress-canary-shx9t\" (UID: \"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7\") " pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.578358 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.578339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9s22\" (UniqueName: \"kubernetes.io/projected/33290017-2b75-42ad-be09-29ee5b7e746e-kube-api-access-k9s22\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:30:59.617497 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.617478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-shx9t" Sep 11 11:30:59.894515 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.894482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:30:59.894670 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.894482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:30:59.894670 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.894621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:30:59.897554 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.897525 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 11:30:59.897694 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.897667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 11:30:59.897818 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.897774 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcr2x\"" Sep 11 11:30:59.897818 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.897775 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 11:30:59.897926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.897820 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 11:30:59.897926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.897776 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgxpf\"" Sep 11 11:30:59.943377 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.943356 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-shx9t"] Sep 11 11:30:59.946580 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:30:59.946560 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc8c5a1_ddea_4819_909d_23aae4d3e4f7.slice/crio-3da43ce1ae71721d4f44c8a2d7f04b25c86778d75e76e917b7799f023b93fab2 WatchSource:0}: Error finding container 3da43ce1ae71721d4f44c8a2d7f04b25c86778d75e76e917b7799f023b93fab2: Status 404 returned error can't find the container with id 3da43ce1ae71721d4f44c8a2d7f04b25c86778d75e76e917b7799f023b93fab2 Sep 11 11:30:59.971082 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:30:59.971034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:30:59.971164 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.971153 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:30:59.971164 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.971163 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5dcd6d57-m697t: secret "image-registry-tls" not found Sep 11 11:30:59.971238 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:30:59.971203 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls podName:9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:00.971191114 +0000 UTC m=+37.507098089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls") pod "image-registry-6f5dcd6d57-m697t" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933") : secret "image-registry-tls" not found Sep 11 11:31:00.071825 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:00.071799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:00.071952 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:00.071935 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:31:00.071996 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:00.071991 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls podName:33290017-2b75-42ad-be09-29ee5b7e746e nodeName:}" failed. No retries permitted until 2026-09-11 11:31:01.071978436 +0000 UTC m=+37.607885411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls") pod "dns-default-6h82f" (UID: "33290017-2b75-42ad-be09-29ee5b7e746e") : secret "dns-default-metrics-tls" not found Sep 11 11:31:00.140576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:00.140548 2579 generic.go:356] "Generic (PLEG): container finished" podID="e09db19d-ecec-41b5-85e3-164979161e08" containerID="cdb0ed4e69c36d1eca1ea4f2b9fe4b1f197624d65c646d18537aa696da32b555" exitCode=0 Sep 11 11:31:00.141212 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:00.140615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerDied","Data":"cdb0ed4e69c36d1eca1ea4f2b9fe4b1f197624d65c646d18537aa696da32b555"} Sep 11 11:31:00.141963 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:00.141944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shx9t" event={"ID":"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7","Type":"ContainerStarted","Data":"3da43ce1ae71721d4f44c8a2d7f04b25c86778d75e76e917b7799f023b93fab2"} Sep 11 11:31:00.978450 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:00.978418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:00.978612 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:00.978593 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:31:00.978662 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:00.978619 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5dcd6d57-m697t: secret "image-registry-tls" not found Sep 11 11:31:00.978738 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:00.978692 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls podName:9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:02.978670215 +0000 UTC m=+39.514577193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls") pod "image-registry-6f5dcd6d57-m697t" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933") : secret "image-registry-tls" not found Sep 11 11:31:01.079415 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:01.079382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:01.079566 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:01.079496 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:31:01.079566 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:01.079546 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls podName:33290017-2b75-42ad-be09-29ee5b7e746e nodeName:}" failed. No retries permitted until 2026-09-11 11:31:03.07953364 +0000 UTC m=+39.615440618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls") pod "dns-default-6h82f" (UID: "33290017-2b75-42ad-be09-29ee5b7e746e") : secret "dns-default-metrics-tls" not found Sep 11 11:31:01.149514 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:01.149481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" event={"ID":"e09db19d-ecec-41b5-85e3-164979161e08","Type":"ContainerStarted","Data":"870cd6f7b2757c2b3d847e13dd9c289286c12621c58879dfa3906233035e2c0f"} Sep 11 11:31:01.173690 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:01.173647 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8kkp9" podStartSLOduration=8.114434041 podStartE2EDuration="37.173634074s" totalImagesPullingTime="29.059200033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:01.17210294 +0000 UTC m=+37.708009940" watchObservedRunningTime="2026-09-11 11:31:01.173634074 +0000 UTC m=+37.709541068" Sep 11 11:31:02.993270 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:02.992965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:02.993270 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:02.993129 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:31:02.993822 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:02.993277 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5dcd6d57-m697t: secret "image-registry-tls" not found Sep 11 11:31:02.993822 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:02.993337 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls podName:9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:06.993320744 +0000 UTC m=+43.529227725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls") pod "image-registry-6f5dcd6d57-m697t" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933") : secret "image-registry-tls" not found Sep 11 11:31:03.094199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:03.094162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:03.094350 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:03.094281 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:31:03.094350 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:03.094337 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls podName:33290017-2b75-42ad-be09-29ee5b7e746e nodeName:}" failed. No retries permitted until 2026-09-11 11:31:07.09432219 +0000 UTC m=+43.630229165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls") pod "dns-default-6h82f" (UID: "33290017-2b75-42ad-be09-29ee5b7e746e") : secret "dns-default-metrics-tls" not found Sep 11 11:31:03.156501 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:03.156466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-shx9t" event={"ID":"bbc8c5a1-ddea-4819-909d-23aae4d3e4f7","Type":"ContainerStarted","Data":"112f6c5327a832c2a3dc30ab1407e326f84ab26d94f7da8ba832abef694da035"} Sep 11 11:31:03.172241 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:03.172195 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-shx9t" podStartSLOduration=1.4105899800000001 podStartE2EDuration="4.172179581s" totalImagesPullingTime="2.761589601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:03.17127289 +0000 UTC m=+39.707179888" watchObservedRunningTime="2026-09-11 11:31:03.172179581 +0000 UTC m=+39.708086578" Sep 11 11:31:04.207322 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.207290 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zrnsq"] Sep 11 11:31:04.244604 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.244562 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zrnsq"] Sep 11 11:31:04.244760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.244680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.247828 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.247705 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 11:31:04.248401 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.248133 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 11:31:04.248401 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.247806 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 11:31:04.248401 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.247866 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75kbm\"" Sep 11 11:31:04.251310 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.248688 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 11:31:04.404826 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.404788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.404826 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.404833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18b1ca60-6452-4e15-a673-3be9f7132d64-data-volume\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.405035 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.404893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18b1ca60-6452-4e15-a673-3be9f7132d64-crio-socket\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.405035 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.404908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.405035 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.404937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcvr\" (UniqueName: \"kubernetes.io/projected/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-api-access-ftcvr\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.505360 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.505283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18b1ca60-6452-4e15-a673-3be9f7132d64-data-volume\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.505360 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.505348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18b1ca60-6452-4e15-a673-3be9f7132d64-crio-socket\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.505526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.505387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.505526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.505429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcvr\" (UniqueName: \"kubernetes.io/projected/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-api-access-ftcvr\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.505526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.505477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.505526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.505494 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18b1ca60-6452-4e15-a673-3be9f7132d64-crio-socket\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.505526 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:04.505501 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:31:04.505695 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:04.505583 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:31:04.505695 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:04.505588 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:05.005566615 +0000 UTC m=+41.541473611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : configmap "kube-rbac-proxy" not found Sep 11 11:31:04.505695 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:04.505644 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:05.005626726 +0000 UTC m=+41.541533701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : secret "insights-runtime-extractor-tls" not found Sep 11 11:31:04.505695 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.505673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18b1ca60-6452-4e15-a673-3be9f7132d64-data-volume\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:04.515087 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:04.515061 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcvr\" (UniqueName: \"kubernetes.io/projected/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-api-access-ftcvr\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:05.009588 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:05.009556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:05.009762 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:05.009625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:05.009762 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:05.009705 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:31:05.009762 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:05.009751 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:31:05.009870 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:05.009780 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:06.009764843 +0000 UTC m=+42.545671818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : secret "insights-runtime-extractor-tls" not found Sep 11 11:31:05.009870 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:05.009797 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:06.00978611 +0000 UTC m=+42.545693085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : configmap "kube-rbac-proxy" not found Sep 11 11:31:06.017487 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:06.017458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:06.017913 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:06.017514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:06.017913 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:06.017571 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:31:06.017913 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:06.017626 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:31:06.017913 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:06.017633 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:08.017618715 +0000 UTC m=+44.553525718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : configmap "kube-rbac-proxy" not found Sep 11 11:31:06.017913 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:06.017665 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:08.017654182 +0000 UTC m=+44.553561156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : secret "insights-runtime-extractor-tls" not found Sep 11 11:31:06.171103 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:06.171072 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fzqcl_239283ff-d4c3-469a-b910-44d112a3c32a/dns-node-resolver/0.log" Sep 11 11:31:06.972132 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:06.972096 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4s48h_3499a298-490e-489b-a5fb-307cd8fcf6e9/node-ca/0.log" Sep 11 11:31:07.023775 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:07.023749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:07.024109 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:07.023873 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:31:07.024109 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:07.023885 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5dcd6d57-m697t: secret "image-registry-tls" not found Sep 11 11:31:07.024109 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:07.023930 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls podName:9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:15.023916613 +0000 UTC m=+51.559823588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls") pod "image-registry-6f5dcd6d57-m697t" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933") : secret "image-registry-tls" not found Sep 11 11:31:07.124818 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:07.124784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:07.124953 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:07.124935 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:31:07.125002 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:07.124991 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls podName:33290017-2b75-42ad-be09-29ee5b7e746e nodeName:}" failed. No retries permitted until 2026-09-11 11:31:15.124974866 +0000 UTC m=+51.660881842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls") pod "dns-default-6h82f" (UID: "33290017-2b75-42ad-be09-29ee5b7e746e") : secret "dns-default-metrics-tls" not found Sep 11 11:31:08.030953 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:08.030920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:08.031309 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:08.031007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:08.031309 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:08.031074 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:31:08.031309 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:08.031120 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:31:08.031309 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:08.031133 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:12.031120465 +0000 UTC m=+48.567027440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : secret "insights-runtime-extractor-tls" not found Sep 11 11:31:08.031309 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:08.031169 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:12.031156735 +0000 UTC m=+48.567063732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : configmap "kube-rbac-proxy" not found Sep 11 11:31:08.171505 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:08.171478 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shx9t_bbc8c5a1-ddea-4819-909d-23aae4d3e4f7/serve-healthcheck-canary/0.log" Sep 11 11:31:12.062053 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:12.062015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:12.062465 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:12.062112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:12.062465 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:12.062161 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:31:12.062465 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:12.062239 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls podName:18b1ca60-6452-4e15-a673-3be9f7132d64 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:20.062217135 +0000 UTC m=+56.598124109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zrnsq" (UID: "18b1ca60-6452-4e15-a673-3be9f7132d64") : secret "insights-runtime-extractor-tls" not found Sep 11 11:31:12.062648 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:12.062628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18b1ca60-6452-4e15-a673-3be9f7132d64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:15.081804 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:15.081772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:15.082275 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:15.081920 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:31:15.082275 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:15.081936 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5dcd6d57-m697t: secret "image-registry-tls" not found Sep 11 11:31:15.082275 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:15.081985 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls podName:9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933 nodeName:}" failed. No retries permitted until 2026-09-11 11:31:31.081969001 +0000 UTC m=+67.617875978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls") pod "image-registry-6f5dcd6d57-m697t" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933") : secret "image-registry-tls" not found Sep 11 11:31:15.182402 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:15.182373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:15.184823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:15.184799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/33290017-2b75-42ad-be09-29ee5b7e746e-metrics-tls\") pod \"dns-default-6h82f\" (UID: \"33290017-2b75-42ad-be09-29ee5b7e746e\") " pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:15.291363 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:15.291334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:15.570797 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:15.570769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6h82f"] Sep 11 11:31:15.573838 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:15.573811 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33290017_2b75_42ad_be09_29ee5b7e746e.slice/crio-10588d667b095436245a1493f2a1d9f193ebb7cfa2b5ee1ed4bbcde780598154 WatchSource:0}: Error finding container 10588d667b095436245a1493f2a1d9f193ebb7cfa2b5ee1ed4bbcde780598154: Status 404 returned error can't find the container with id 10588d667b095436245a1493f2a1d9f193ebb7cfa2b5ee1ed4bbcde780598154 Sep 11 11:31:16.185816 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:16.185779 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6h82f" event={"ID":"33290017-2b75-42ad-be09-29ee5b7e746e","Type":"ContainerStarted","Data":"10588d667b095436245a1493f2a1d9f193ebb7cfa2b5ee1ed4bbcde780598154"} Sep 11 11:31:18.193139 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:18.193103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6h82f" event={"ID":"33290017-2b75-42ad-be09-29ee5b7e746e","Type":"ContainerStarted","Data":"a785d0ebe79f24aedff45e313c66a29179b61280eeb5c6b1e841e37e539e6a4c"} Sep 11 11:31:18.193581 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:18.193154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6h82f" event={"ID":"33290017-2b75-42ad-be09-29ee5b7e746e","Type":"ContainerStarted","Data":"39cd2bdec395270c4039e29bdd4eaf70c36a113b6329077a2d96cfc3d7db0bc1"} Sep 11 11:31:18.193581 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:18.193240 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:18.210060 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:18.210009 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6h82f" podStartSLOduration=17.443779506 podStartE2EDuration="19.209993279s" totalImagesPullingTime="1.766213773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:18.209432679 +0000 UTC m=+54.745339688" watchObservedRunningTime="2026-09-11 11:31:18.209993279 +0000 UTC m=+54.745900271" Sep 11 11:31:20.118953 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:20.118913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:20.121326 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:20.121304 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18b1ca60-6452-4e15-a673-3be9f7132d64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrnsq\" (UID: \"18b1ca60-6452-4e15-a673-3be9f7132d64\") " pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:20.155348 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:20.155323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zrnsq" Sep 11 11:31:20.420759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:20.420658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zrnsq"] Sep 11 11:31:20.423436 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:20.423405 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b1ca60_6452_4e15_a673_3be9f7132d64.slice/crio-24b4fe8f6cf7b3e8e935d9a666e54242731117448973a800273185df6a72d343 WatchSource:0}: Error finding container 24b4fe8f6cf7b3e8e935d9a666e54242731117448973a800273185df6a72d343: Status 404 returned error can't find the container with id 24b4fe8f6cf7b3e8e935d9a666e54242731117448973a800273185df6a72d343 Sep 11 11:31:21.203170 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:21.203137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrnsq" event={"ID":"18b1ca60-6452-4e15-a673-3be9f7132d64","Type":"ContainerStarted","Data":"f96d4c3479869fb36e0c6e4faa57193d31b78de06692f4ce4a513524a89be81b"} Sep 11 11:31:21.203512 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:21.203199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrnsq" event={"ID":"18b1ca60-6452-4e15-a673-3be9f7132d64","Type":"ContainerStarted","Data":"24b4fe8f6cf7b3e8e935d9a666e54242731117448973a800273185df6a72d343"} Sep 11 11:31:22.207161 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:22.207127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrnsq" event={"ID":"18b1ca60-6452-4e15-a673-3be9f7132d64","Type":"ContainerStarted","Data":"b0d691421410343ed09aeb009bdf1f20ec67753cbece10f129a79da375682a09"} Sep 11 11:31:24.132591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:24.132567 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zcd47" Sep 11 11:31:24.214265 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:24.214238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrnsq" event={"ID":"18b1ca60-6452-4e15-a673-3be9f7132d64","Type":"ContainerStarted","Data":"32d0ec69fd4f388e01221031c25d0d39a070a9c536d142dbd6cba9ec8eace2a2"} Sep 11 11:31:24.233438 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:24.233396 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zrnsq" podStartSLOduration=17.571051873000002 podStartE2EDuration="20.233381421s" totalImagesPullingTime="2.662329548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:31:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:24.232203395 +0000 UTC m=+60.768110392" watchObservedRunningTime="2026-09-11 11:31:24.233381421 +0000 UTC m=+60.769288415" Sep 11 11:31:28.198035 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:28.198008 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6h82f" Sep 11 11:31:29.586222 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.586187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:31:29.588963 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.588945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 11:31:29.598924 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.598905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/222b1b29-9ac9-4418-b6c2-271ff86e4850-original-pull-secret\") pod \"global-pull-secret-syncer-d6f7p\" (UID: \"222b1b29-9ac9-4418-b6c2-271ff86e4850\") " pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:31:29.604869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.604849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6f7p" Sep 11 11:31:29.686977 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.686945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:31:29.689961 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.689939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 11:31:29.710610 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.710586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eafd60c-bb2d-4473-ab8a-76180f836548-metrics-certs\") pod \"network-metrics-daemon-59gwj\" (UID: \"7eafd60c-bb2d-4473-ab8a-76180f836548\") " pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:31:29.787902 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.787871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:31:29.790794 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.790778 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 11:31:29.800969 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.800951 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 11:31:29.810415 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.810393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldh6\" (UniqueName: \"kubernetes.io/projected/ba4cf6d4-6127-4838-8e66-f71984a95365-kube-api-access-jldh6\") pod \"network-check-target-prvf7\" (UID: \"ba4cf6d4-6127-4838-8e66-f71984a95365\") " pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:31:29.913130 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.913112 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgxpf\"" Sep 11 11:31:29.916612 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.916596 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zcr2x\"" Sep 11 11:31:29.920916 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.920898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-59gwj" Sep 11 11:31:29.924472 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.924417 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6f7p"] Sep 11 11:31:29.924472 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.924444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:31:29.929911 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:29.929882 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222b1b29_9ac9_4418_b6c2_271ff86e4850.slice/crio-c089032c583fbf5239c03007232ad7362e4f339ccc17e9a58b0e480c55a3683e WatchSource:0}: Error finding container c089032c583fbf5239c03007232ad7362e4f339ccc17e9a58b0e480c55a3683e: Status 404 returned error can't find the container with id c089032c583fbf5239c03007232ad7362e4f339ccc17e9a58b0e480c55a3683e Sep 11 11:31:29.955472 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.955445 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r"] Sep 11 11:31:29.958386 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.958356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:29.961512 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.961481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 11:31:29.961512 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.961496 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 11:31:29.961815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.961796 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 11:31:29.961996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.961976 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 11:31:29.966154 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.966086 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z"] Sep 11 11:31:29.974551 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.972487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:29.976393 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.975958 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r"] Sep 11 11:31:29.976529 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.976456 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 11:31:29.976598 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:29.976534 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-p2vjk\"" Sep 11 11:31:30.013210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.012366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6g6\" (UniqueName: \"kubernetes.io/projected/ecefa57a-457c-478a-9b03-c40e9537873d-kube-api-access-bt6g6\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.013210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.012445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecefa57a-457c-478a-9b03-c40e9537873d-tmp\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.013210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.012471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ecefa57a-457c-478a-9b03-c40e9537873d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.013210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.012516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c99f0882-2f9f-453e-a821-732efce1c9ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6874794948-qhn9z\" (UID: \"c99f0882-2f9f-453e-a821-732efce1c9ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:30.013210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.012541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fbj\" (UniqueName: \"kubernetes.io/projected/c99f0882-2f9f-453e-a821-732efce1c9ef-kube-api-access-v9fbj\") pod \"managed-serviceaccount-addon-agent-6874794948-qhn9z\" (UID: \"c99f0882-2f9f-453e-a821-732efce1c9ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:30.018837 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.014808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z"] Sep 11 11:31:30.079051 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.079022 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-mhwfg"] Sep 11 11:31:30.080944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.080929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-mhwfg" Sep 11 11:31:30.084849 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.084830 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kpfb7\"" Sep 11 11:31:30.086521 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.086498 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 11:31:30.091150 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.091133 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 11:31:30.113287 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.113260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-mhwfg"] Sep 11 11:31:30.113459 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.113438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6g6\" (UniqueName: \"kubernetes.io/projected/ecefa57a-457c-478a-9b03-c40e9537873d-kube-api-access-bt6g6\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.113527 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.113474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twzfh\" (UniqueName: \"kubernetes.io/projected/0f79254d-a7c1-40ce-a08c-678276ff68ac-kube-api-access-twzfh\") pod \"downloads-76f99584f4-mhwfg\" (UID: \"0f79254d-a7c1-40ce-a08c-678276ff68ac\") " pod="openshift-console/downloads-76f99584f4-mhwfg" Sep 11 11:31:30.113527 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.113523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecefa57a-457c-478a-9b03-c40e9537873d-tmp\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.113639 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.113546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ecefa57a-457c-478a-9b03-c40e9537873d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.113639 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.113573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c99f0882-2f9f-453e-a821-732efce1c9ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6874794948-qhn9z\" (UID: \"c99f0882-2f9f-453e-a821-732efce1c9ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:30.113639 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.113592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fbj\" (UniqueName: \"kubernetes.io/projected/c99f0882-2f9f-453e-a821-732efce1c9ef-kube-api-access-v9fbj\") pod \"managed-serviceaccount-addon-agent-6874794948-qhn9z\" (UID: \"c99f0882-2f9f-453e-a821-732efce1c9ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:30.114035 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.114016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ecefa57a-457c-478a-9b03-c40e9537873d-tmp\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.116182 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.116161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c99f0882-2f9f-453e-a821-732efce1c9ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6874794948-qhn9z\" (UID: \"c99f0882-2f9f-453e-a821-732efce1c9ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:30.116261 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.116234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ecefa57a-457c-478a-9b03-c40e9537873d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.131937 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.131917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fbj\" (UniqueName: \"kubernetes.io/projected/c99f0882-2f9f-453e-a821-732efce1c9ef-kube-api-access-v9fbj\") pod \"managed-serviceaccount-addon-agent-6874794948-qhn9z\" (UID: \"c99f0882-2f9f-453e-a821-732efce1c9ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:30.138078 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.138056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6g6\" (UniqueName: \"kubernetes.io/projected/ecefa57a-457c-478a-9b03-c40e9537873d-kube-api-access-bt6g6\") pod \"klusterlet-addon-workmgr-6b5d878674-jsl6r\" (UID: \"ecefa57a-457c-478a-9b03-c40e9537873d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.214038 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.213988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twzfh\" (UniqueName: \"kubernetes.io/projected/0f79254d-a7c1-40ce-a08c-678276ff68ac-kube-api-access-twzfh\") pod \"downloads-76f99584f4-mhwfg\" (UID: \"0f79254d-a7c1-40ce-a08c-678276ff68ac\") " pod="openshift-console/downloads-76f99584f4-mhwfg" Sep 11 11:31:30.220821 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.220800 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-59gwj"] Sep 11 11:31:30.223497 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.223379 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-prvf7"] Sep 11 11:31:30.223760 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:30.223739 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eafd60c_bb2d_4473_ab8a_76180f836548.slice/crio-9ad2cb224a496ba3fbad6a2461ed012a97de03359f31357b6b744afbe2f6f704 WatchSource:0}: Error finding container 9ad2cb224a496ba3fbad6a2461ed012a97de03359f31357b6b744afbe2f6f704: Status 404 returned error can't find the container with id 9ad2cb224a496ba3fbad6a2461ed012a97de03359f31357b6b744afbe2f6f704 Sep 11 11:31:30.225700 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:30.225680 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4cf6d4_6127_4838_8e66_f71984a95365.slice/crio-6ab104a5258daafc88f0aadaedee6f99f060c6a0ceb830837144c09160f445fc WatchSource:0}: Error finding container 6ab104a5258daafc88f0aadaedee6f99f060c6a0ceb830837144c09160f445fc: Status 404 returned error can't find the container with id 6ab104a5258daafc88f0aadaedee6f99f060c6a0ceb830837144c09160f445fc Sep 11 11:31:30.228960 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.228945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twzfh\" (UniqueName: \"kubernetes.io/projected/0f79254d-a7c1-40ce-a08c-678276ff68ac-kube-api-access-twzfh\") pod \"downloads-76f99584f4-mhwfg\" (UID: \"0f79254d-a7c1-40ce-a08c-678276ff68ac\") " pod="openshift-console/downloads-76f99584f4-mhwfg" Sep 11 11:31:30.232996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.232974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d6f7p" event={"ID":"222b1b29-9ac9-4418-b6c2-271ff86e4850","Type":"ContainerStarted","Data":"c089032c583fbf5239c03007232ad7362e4f339ccc17e9a58b0e480c55a3683e"} Sep 11 11:31:30.234032 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.234012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-59gwj" event={"ID":"7eafd60c-bb2d-4473-ab8a-76180f836548","Type":"ContainerStarted","Data":"9ad2cb224a496ba3fbad6a2461ed012a97de03359f31357b6b744afbe2f6f704"} Sep 11 11:31:30.235121 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.235103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-prvf7" event={"ID":"ba4cf6d4-6127-4838-8e66-f71984a95365","Type":"ContainerStarted","Data":"6ab104a5258daafc88f0aadaedee6f99f060c6a0ceb830837144c09160f445fc"} Sep 11 11:31:30.275365 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.275342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:30.313800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.313776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" Sep 11 11:31:30.390009 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.389983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-mhwfg" Sep 11 11:31:30.539787 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.539763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r"] Sep 11 11:31:30.542054 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:30.542033 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecefa57a_457c_478a_9b03_c40e9537873d.slice/crio-eb3c137b9a6fcb677e1b7cc1f5100ade884e33dc93703ab1080f87baf0c4cb0d WatchSource:0}: Error finding container eb3c137b9a6fcb677e1b7cc1f5100ade884e33dc93703ab1080f87baf0c4cb0d: Status 404 returned error can't find the container with id eb3c137b9a6fcb677e1b7cc1f5100ade884e33dc93703ab1080f87baf0c4cb0d Sep 11 11:31:30.580687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.580660 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z"] Sep 11 11:31:30.583662 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:30.583639 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc99f0882_2f9f_453e_a821_732efce1c9ef.slice/crio-1a3b3c74b24571e965eb3a8ddb2d3bbf0004d2a685be9710aac90f84acd7f5d6 WatchSource:0}: Error finding container 1a3b3c74b24571e965eb3a8ddb2d3bbf0004d2a685be9710aac90f84acd7f5d6: Status 404 returned error can't find the container with id 1a3b3c74b24571e965eb3a8ddb2d3bbf0004d2a685be9710aac90f84acd7f5d6 Sep 11 11:31:30.656046 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:30.655870 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-mhwfg"] Sep 11 11:31:30.658129 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:30.658100 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f79254d_a7c1_40ce_a08c_678276ff68ac.slice/crio-a00d32461505051a1f8b08934bc920267e1c4cf476d169e0c3c3375896991a96 WatchSource:0}: Error finding container a00d32461505051a1f8b08934bc920267e1c4cf476d169e0c3c3375896991a96: Status 404 returned error can't find the container with id a00d32461505051a1f8b08934bc920267e1c4cf476d169e0c3c3375896991a96 Sep 11 11:31:31.121381 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.121342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:31.131555 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.131527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"image-registry-6f5dcd6d57-m697t\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:31.242095 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.242055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-mhwfg" event={"ID":"0f79254d-a7c1-40ce-a08c-678276ff68ac","Type":"ContainerStarted","Data":"a00d32461505051a1f8b08934bc920267e1c4cf476d169e0c3c3375896991a96"} Sep 11 11:31:31.243705 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.243673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" event={"ID":"ecefa57a-457c-478a-9b03-c40e9537873d","Type":"ContainerStarted","Data":"eb3c137b9a6fcb677e1b7cc1f5100ade884e33dc93703ab1080f87baf0c4cb0d"} Sep 11 11:31:31.245634 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.245608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" event={"ID":"c99f0882-2f9f-453e-a821-732efce1c9ef","Type":"ContainerStarted","Data":"1a3b3c74b24571e965eb3a8ddb2d3bbf0004d2a685be9710aac90f84acd7f5d6"} Sep 11 11:31:31.392997 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.392972 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mqz7m\"" Sep 11 11:31:31.401045 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.400645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:31.670165 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.669217 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9"] Sep 11 11:31:31.674516 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.674109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" Sep 11 11:31:31.676805 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.676785 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 11:31:31.677064 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.677047 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5kqjk\"" Sep 11 11:31:31.683267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.683211 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9"] Sep 11 11:31:31.725580 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.725553 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f5dcd6d57-m697t"] Sep 11 11:31:31.725931 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.725854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/851078be-7141-4281-87c1-7d78c25b3ab4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ntdz9\" (UID: \"851078be-7141-4281-87c1-7d78c25b3ab4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" Sep 11 11:31:31.730803 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:31.730776 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4061b8_cd8f_4c1c_8b97_dcf08b1bc933.slice/crio-4e474c8d793af80bc9e3234a1a5c4ce1c94059e672571b049509bd328f543d42 WatchSource:0}: Error finding container 4e474c8d793af80bc9e3234a1a5c4ce1c94059e672571b049509bd328f543d42: Status 404 returned error can't find the container with id 4e474c8d793af80bc9e3234a1a5c4ce1c94059e672571b049509bd328f543d42 Sep 11 11:31:31.826853 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.826830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/851078be-7141-4281-87c1-7d78c25b3ab4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ntdz9\" (UID: \"851078be-7141-4281-87c1-7d78c25b3ab4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" Sep 11 11:31:31.830647 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.830623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/851078be-7141-4281-87c1-7d78c25b3ab4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-ntdz9\" (UID: \"851078be-7141-4281-87c1-7d78c25b3ab4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" Sep 11 11:31:31.904283 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.902281 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c8cb75595-dddn4"] Sep 11 11:31:31.907168 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.907142 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8cb75595-dddn4"] Sep 11 11:31:31.907344 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.907257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:31.911805 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.911781 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 11:31:31.912058 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.912038 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 11:31:31.912194 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.912181 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 11:31:31.912763 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.912256 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 11:31:31.912763 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.912372 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 11:31:31.912763 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.912517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cs8hc\"" Sep 11 11:31:31.928097 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.928071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-serving-cert\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:31.928238 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.928224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-oauth-config\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:31.928367 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.928352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4ch\" (UniqueName: \"kubernetes.io/projected/df247472-9d6b-4bf4-a1bb-615ca1312714-kube-api-access-jd4ch\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:31.928486 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.928472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-service-ca\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:31.928631 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.928618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-console-config\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:31.928772 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.928759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-oauth-serving-cert\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:31.987616 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:31.987586 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" Sep 11 11:31:32.030382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.029408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-serving-cert\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.030382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.029456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-oauth-config\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.030382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.029491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4ch\" (UniqueName: \"kubernetes.io/projected/df247472-9d6b-4bf4-a1bb-615ca1312714-kube-api-access-jd4ch\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.030382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.029527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-service-ca\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.030382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.029558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-console-config\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.030382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.029580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-oauth-serving-cert\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.032260 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.031546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-oauth-serving-cert\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.032260 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.032181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-service-ca\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.032260 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.032187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-console-config\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.035657 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.035634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-oauth-config\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.040400 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.040374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-serving-cert\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.045144 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.045099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4ch\" (UniqueName: \"kubernetes.io/projected/df247472-9d6b-4bf4-a1bb-615ca1312714-kube-api-access-jd4ch\") pod \"console-6c8cb75595-dddn4\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.226579 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.226281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:32.256433 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.255640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-59gwj" event={"ID":"7eafd60c-bb2d-4473-ab8a-76180f836548","Type":"ContainerStarted","Data":"08ba9570c4cbd8879107fcdcdf15685ba23b115924747cf3ec1320438b38050c"} Sep 11 11:31:32.256433 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.255704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-59gwj" event={"ID":"7eafd60c-bb2d-4473-ab8a-76180f836548","Type":"ContainerStarted","Data":"b8a27016131381f4dc25e35254ba740187f4c7804404bee1b2f2cdeab7fb2734"} Sep 11 11:31:32.260580 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.260096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" event={"ID":"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933","Type":"ContainerStarted","Data":"1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032"} Sep 11 11:31:32.260580 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.260143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" event={"ID":"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933","Type":"ContainerStarted","Data":"4e474c8d793af80bc9e3234a1a5c4ce1c94059e672571b049509bd328f543d42"} Sep 11 11:31:32.260796 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.260607 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:32.273995 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.273946 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-59gwj" podStartSLOduration=67.186138512 podStartE2EDuration="1m8.273928s" totalImagesPullingTime="1.087789488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:32.273504805 +0000 UTC m=+68.809411803" watchObservedRunningTime="2026-09-11 11:31:32.273928 +0000 UTC m=+68.809834994" Sep 11 11:31:32.335996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.331404 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" podStartSLOduration=62.331379355 podStartE2EDuration="1m2.331379355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:32.296433386 +0000 UTC m=+68.832340385" watchObservedRunningTime="2026-09-11 11:31:32.331379355 +0000 UTC m=+68.867286356" Sep 11 11:31:32.335996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.331698 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9"] Sep 11 11:31:32.560780 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:32.560663 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c8cb75595-dddn4"] Sep 11 11:31:32.563575 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:32.563530 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf247472_9d6b_4bf4_a1bb_615ca1312714.slice/crio-b819e88440509109e1e69c3a227deb9389dcea75b872960599470a20acd5315e WatchSource:0}: Error finding container b819e88440509109e1e69c3a227deb9389dcea75b872960599470a20acd5315e: Status 404 returned error can't find the container with id b819e88440509109e1e69c3a227deb9389dcea75b872960599470a20acd5315e Sep 11 11:31:33.268433 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:33.268396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8cb75595-dddn4" event={"ID":"df247472-9d6b-4bf4-a1bb-615ca1312714","Type":"ContainerStarted","Data":"b819e88440509109e1e69c3a227deb9389dcea75b872960599470a20acd5315e"} Sep 11 11:31:33.272435 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:33.272403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" event={"ID":"851078be-7141-4281-87c1-7d78c25b3ab4","Type":"ContainerStarted","Data":"2501eb34301af19a6e5792cfb0671505a5393477759ac7bc982fc21a362b088b"} Sep 11 11:31:34.797147 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.797113 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-796789d579-gfgvw"] Sep 11 11:31:34.800765 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.800574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.811787 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.810822 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 11:31:34.812568 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.812544 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796789d579-gfgvw"] Sep 11 11:31:34.856841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.856573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-service-ca\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.856841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.856638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htn9w\" (UniqueName: \"kubernetes.io/projected/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-kube-api-access-htn9w\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.856841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.856673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-trusted-ca-bundle\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.856841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.856737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-oauth-serving-cert\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.856841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.856782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-serving-cert\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.856841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.856810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-oauth-config\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.857261 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.856891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-config\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.958226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-service-ca\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.958289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htn9w\" (UniqueName: \"kubernetes.io/projected/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-kube-api-access-htn9w\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.958321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-trusted-ca-bundle\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.958360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-oauth-serving-cert\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.958402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-serving-cert\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.958434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-oauth-config\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.958469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-config\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.959192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-config\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.959800 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.959745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-service-ca\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.960511 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.960168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-oauth-serving-cert\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.960575 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.960552 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-trusted-ca-bundle\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.962597 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.962546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-oauth-config\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.963454 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.963429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-serving-cert\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:34.968160 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:34.968118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htn9w\" (UniqueName: \"kubernetes.io/projected/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-kube-api-access-htn9w\") pod \"console-796789d579-gfgvw\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:35.119372 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:35.119335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:39.304310 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.304270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" event={"ID":"c99f0882-2f9f-453e-a821-732efce1c9ef","Type":"ContainerStarted","Data":"605576feba578f53f8217095d55c911ed805a8262fa675d6e5f5679527b139ec"} Sep 11 11:31:39.308410 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.307871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8cb75595-dddn4" event={"ID":"df247472-9d6b-4bf4-a1bb-615ca1312714","Type":"ContainerStarted","Data":"b9cea677f6d675bccea50c3b848c3ab1af3103df4f290ddf611b93767a9a08e3"} Sep 11 11:31:39.310673 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.310361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" event={"ID":"851078be-7141-4281-87c1-7d78c25b3ab4","Type":"ContainerStarted","Data":"123d241c88d6720a3a687d44e578883e3792fa8bf88b5ee908d606fb3ca97402"} Sep 11 11:31:39.310673 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.310526 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" Sep 11 11:31:39.314908 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.314877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-prvf7" event={"ID":"ba4cf6d4-6127-4838-8e66-f71984a95365","Type":"ContainerStarted","Data":"058730bd3986ece01797177a071277a01f7c930101ce655912868b4afdcabcff"} Sep 11 11:31:39.315309 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.315290 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:31:39.323141 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.321815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" Sep 11 11:31:39.323256 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.323182 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6874794948-qhn9z" podStartSLOduration=1.80065348 podStartE2EDuration="10.323166956s" totalImagesPullingTime="8.522513476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:39.321290703 +0000 UTC m=+75.857197699" watchObservedRunningTime="2026-09-11 11:31:39.323166956 +0000 UTC m=+75.859073953" Sep 11 11:31:39.339900 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.338768 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c8cb75595-dddn4" podStartSLOduration=1.7227736999999999 podStartE2EDuration="8.338749155s" totalImagesPullingTime="6.615975455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:39.337967544 +0000 UTC m=+75.873874540" watchObservedRunningTime="2026-09-11 11:31:39.338749155 +0000 UTC m=+75.874656153" Sep 11 11:31:39.355126 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.354987 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-ntdz9" podStartSLOduration=1.603320182 podStartE2EDuration="8.354967709s" totalImagesPullingTime="6.751647527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:39.353232198 +0000 UTC m=+75.889139195" watchObservedRunningTime="2026-09-11 11:31:39.354967709 +0000 UTC m=+75.890874711" Sep 11 11:31:39.376340 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.375576 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-prvf7" podStartSLOduration=66.494893053 podStartE2EDuration="1m15.375556961s" totalImagesPullingTime="8.880663908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:39.374281946 +0000 UTC m=+75.910188944" watchObservedRunningTime="2026-09-11 11:31:39.375556961 +0000 UTC m=+75.911463958" Sep 11 11:31:39.385032 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:39.384836 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-796789d579-gfgvw"] Sep 11 11:31:39.388420 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:39.388384 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f5dcd7_0c1c_4994_b0be_837d04d1c1bd.slice/crio-ea525c48f39c66a0ef9dd3689cfd66997f98351e986c8d39af0381e4fb6c5dce WatchSource:0}: Error finding container ea525c48f39c66a0ef9dd3689cfd66997f98351e986c8d39af0381e4fb6c5dce: Status 404 returned error can't find the container with id ea525c48f39c66a0ef9dd3689cfd66997f98351e986c8d39af0381e4fb6c5dce Sep 11 11:31:40.322743 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:40.322337 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d6f7p" event={"ID":"222b1b29-9ac9-4418-b6c2-271ff86e4850","Type":"ContainerStarted","Data":"ddf87595f4455e596a2ea763917b6d4d582fe6bf604be108993ced8bbba7fc1b"} Sep 11 11:31:40.326755 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:40.326512 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796789d579-gfgvw" event={"ID":"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd","Type":"ContainerStarted","Data":"a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37"} Sep 11 11:31:40.326755 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:40.326578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796789d579-gfgvw" event={"ID":"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd","Type":"ContainerStarted","Data":"ea525c48f39c66a0ef9dd3689cfd66997f98351e986c8d39af0381e4fb6c5dce"} Sep 11 11:31:40.344475 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:40.343845 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d6f7p" podStartSLOduration=67.166589236 podStartE2EDuration="1m16.343825526s" totalImagesPullingTime="9.17723629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:40.342486242 +0000 UTC m=+76.878393239" watchObservedRunningTime="2026-09-11 11:31:40.343825526 +0000 UTC m=+76.879732519" Sep 11 11:31:40.362786 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:40.362667 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-796789d579-gfgvw" podStartSLOduration=6.362647413 podStartE2EDuration="6.362647413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:40.361800883 +0000 UTC m=+76.897707884" watchObservedRunningTime="2026-09-11 11:31:40.362647413 +0000 UTC m=+76.898554413" Sep 11 11:31:42.227418 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:42.226687 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:42.227418 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:42.227362 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:42.232607 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:42.232446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:42.338056 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:42.338024 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:31:45.120325 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:45.120286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:45.120851 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:45.120396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:45.125960 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:45.125938 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:45.347143 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:45.347117 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:31:45.396904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:45.396830 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c8cb75595-dddn4"] Sep 11 11:31:46.134698 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.134660 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8js97"] Sep 11 11:31:46.141506 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.141475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.142418 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.142391 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n2qz5"] Sep 11 11:31:46.145567 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.145547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.145831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.145809 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 11:31:46.145929 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.145910 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bww2t\"" Sep 11 11:31:46.146072 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.146058 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 11:31:46.146113 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.145809 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 11:31:46.146172 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.146158 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 11:31:46.146240 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.146225 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 11:31:46.146274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.146240 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 11:31:46.148256 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.148236 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 11:31:46.148494 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.148481 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c7mcw\"" Sep 11 11:31:46.148755 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.148645 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 11:31:46.149121 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.148954 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 11:31:46.149209 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.149133 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8js97"] Sep 11 11:31:46.257846 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.257806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.257846 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.257853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-sys\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258065 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.257880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-textfile\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258065 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.257972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258065 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-root\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258065 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f2da0-ee51-463b-8913-b63d884aeec0-metrics-client-ca\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258065 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9f4g\" (UniqueName: \"kubernetes.io/projected/4c4f2da0-ee51-463b-8913-b63d884aeec0-kube-api-access-s9f4g\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f2a1a95-d84c-42e0-8a94-a0571411210b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.258274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.258274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtqw\" (UniqueName: \"kubernetes.io/projected/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-api-access-cxtqw\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.258274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-wtmp\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.258274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f2a1a95-d84c-42e0-8a94-a0571411210b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.258274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.258552 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.258276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-tls\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.359462 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.359462 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-root\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.359687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f2da0-ee51-463b-8913-b63d884aeec0-metrics-client-ca\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.359687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f4g\" (UniqueName: \"kubernetes.io/projected/4c4f2da0-ee51-463b-8913-b63d884aeec0-kube-api-access-s9f4g\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.359687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f2a1a95-d84c-42e0-8a94-a0571411210b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.359687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-root\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.359687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.359687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtqw\" (UniqueName: \"kubernetes.io/projected/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-api-access-cxtqw\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.359687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-wtmp\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f2a1a95-d84c-42e0-8a94-a0571411210b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-tls\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-sys\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-textfile\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.359992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f2a1a95-d84c-42e0-8a94-a0571411210b-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.360481 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.360113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-textfile\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360481 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.360118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360481 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.360186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-sys\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360481 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.360197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f2da0-ee51-463b-8913-b63d884aeec0-metrics-client-ca\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360481 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.360353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-wtmp\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.360750 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.360587 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f2a1a95-d84c-42e0-8a94-a0571411210b-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.361133 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.361084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.364011 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.363949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.364178 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.364156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-tls\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.364517 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.364486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c4f2da0-ee51-463b-8913-b63d884aeec0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.364680 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.364660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.370040 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.370018 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9f4g\" (UniqueName: \"kubernetes.io/projected/4c4f2da0-ee51-463b-8913-b63d884aeec0-kube-api-access-s9f4g\") pod \"node-exporter-n2qz5\" (UID: \"4c4f2da0-ee51-463b-8913-b63d884aeec0\") " pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:46.370429 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.370407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtqw\" (UniqueName: \"kubernetes.io/projected/3f2a1a95-d84c-42e0-8a94-a0571411210b-kube-api-access-cxtqw\") pod \"kube-state-metrics-59fbdd77d-8js97\" (UID: \"3f2a1a95-d84c-42e0-8a94-a0571411210b\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.456103 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.456035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" Sep 11 11:31:46.461875 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:46.461853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2qz5" Sep 11 11:31:47.223731 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.223685 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:31:47.267687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.267658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:31:47.267864 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.267761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.270497 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.270478 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 11:31:47.270649 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.270563 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 11:31:47.270649 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.270567 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 11:31:47.270798 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.270645 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 11:31:47.270948 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.270910 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 11:31:47.271059 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.270995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 11:31:47.271143 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.271125 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 11:31:47.271259 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.271242 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 11:31:47.271389 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.271373 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 11:31:47.271389 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.271386 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q24g9\"" Sep 11 11:31:47.369623 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369623 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369623 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-volume\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369623 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-out\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-web-config\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-kube-api-access-p7bdl\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369909 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.369970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.370306 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.369981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470357 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-volume\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470531 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470531 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470531 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-out\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470531 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470531 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-web-config\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470531 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-kube-api-access-p7bdl\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.470618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.470859 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:47.470768 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle podName:b56095f1-12d4-41e4-bbc0-8f3134678f6e nodeName:}" failed. No retries permitted until 2026-09-11 11:31:47.970744196 +0000 UTC m=+84.506651185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e") : configmap references non-existent config key: ca-bundle.crt Sep 11 11:31:47.471309 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.471276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.471413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.471311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.471413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.471334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.471648 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.471625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.471785 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:47.471743 2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 11 11:31:47.471859 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:47.471787 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls podName:b56095f1-12d4-41e4-bbc0-8f3134678f6e nodeName:}" failed. No retries permitted until 2026-09-11 11:31:47.971772522 +0000 UTC m=+84.507679499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e") : secret "alertmanager-main-tls" not found Sep 11 11:31:47.473705 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.473654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-out\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.473958 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.473913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.474057 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.473980 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-volume\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.474855 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.474836 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.475097 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.475076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.475233 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.475213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.475934 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.475912 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-web-config\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.476779 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.476755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.479731 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.479690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-kube-api-access-p7bdl\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.976267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.976238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.976447 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.976363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.977133 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.977101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:47.979122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:47.979098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:48.179969 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:48.179932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:31:50.300469 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:50.300434 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4f2da0_ee51_463b_8913_b63d884aeec0.slice/crio-2fcd744bedebbad6b82c1ce853ea536744142fe0c90b4ab05146c3dab3411e2f WatchSource:0}: Error finding container 2fcd744bedebbad6b82c1ce853ea536744142fe0c90b4ab05146c3dab3411e2f: Status 404 returned error can't find the container with id 2fcd744bedebbad6b82c1ce853ea536744142fe0c90b4ab05146c3dab3411e2f Sep 11 11:31:50.364027 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.363965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2qz5" event={"ID":"4c4f2da0-ee51-463b-8913-b63d884aeec0","Type":"ContainerStarted","Data":"2fcd744bedebbad6b82c1ce853ea536744142fe0c90b4ab05146c3dab3411e2f"} Sep 11 11:31:50.454040 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.454008 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2"] Sep 11 11:31:50.456890 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.456855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.459868 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.459816 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 11:31:50.460068 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.460047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 11:31:50.460177 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.460160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 11:31:50.461315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.461261 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fa2hsf9k1103i\"" Sep 11 11:31:50.461412 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.461334 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 11:31:50.461619 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.461599 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4gwmr\"" Sep 11 11:31:50.484019 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.483992 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2"] Sep 11 11:31:50.571176 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.571153 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-8js97"] Sep 11 11:31:50.573507 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:50.573476 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2a1a95_d84c_42e0_8a94_a0571411210b.slice/crio-b5cd975c48ff90afd44d92993f145a69791ef900f35bd4a250b59c2f93e1d239 WatchSource:0}: Error finding container b5cd975c48ff90afd44d92993f145a69791ef900f35bd4a250b59c2f93e1d239: Status 404 returned error can't find the container with id b5cd975c48ff90afd44d92993f145a69791ef900f35bd4a250b59c2f93e1d239 Sep 11 11:31:50.580344 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.580298 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:31:50.580762 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:50.580664 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56095f1_12d4_41e4_bbc0_8f3134678f6e.slice/crio-ea745cf05c6d76bf34fd8f76518d250148c47e315a7983a1b2ec6e713d0691b4 WatchSource:0}: Error finding container ea745cf05c6d76bf34fd8f76518d250148c47e315a7983a1b2ec6e713d0691b4: Status 404 returned error can't find the container with id ea745cf05c6d76bf34fd8f76518d250148c47e315a7983a1b2ec6e713d0691b4 Sep 11 11:31:50.600738 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.600695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-secret-metrics-server-tls\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.600843 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.600800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-secret-metrics-server-client-certs\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.600884 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.600843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cfd\" (UniqueName: \"kubernetes.io/projected/59b5b439-d6c7-403c-928a-3815511de21a-kube-api-access-r8cfd\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.600884 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.600867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/59b5b439-d6c7-403c-928a-3815511de21a-audit-log\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.600947 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.600917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/59b5b439-d6c7-403c-928a-3815511de21a-metrics-server-audit-profiles\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.600980 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.600947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b5b439-d6c7-403c-928a-3815511de21a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.601045 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.601021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-client-ca-bundle\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.702075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.702042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-secret-metrics-server-client-certs\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.702251 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.702087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cfd\" (UniqueName: \"kubernetes.io/projected/59b5b439-d6c7-403c-928a-3815511de21a-kube-api-access-r8cfd\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.702251 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.702124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/59b5b439-d6c7-403c-928a-3815511de21a-audit-log\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.702251 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.702152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/59b5b439-d6c7-403c-928a-3815511de21a-metrics-server-audit-profiles\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.702251 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.702195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b5b439-d6c7-403c-928a-3815511de21a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.702251 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.702231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-client-ca-bundle\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.702504 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.702263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-secret-metrics-server-tls\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.703968 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.703422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/59b5b439-d6c7-403c-928a-3815511de21a-audit-log\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.703968 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.703797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/59b5b439-d6c7-403c-928a-3815511de21a-metrics-server-audit-profiles\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.703968 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.703908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59b5b439-d6c7-403c-928a-3815511de21a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.705373 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.705344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-secret-metrics-server-client-certs\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.705922 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.705898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-client-ca-bundle\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.706091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.706073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/59b5b439-d6c7-403c-928a-3815511de21a-secret-metrics-server-tls\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.712808 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.712765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cfd\" (UniqueName: \"kubernetes.io/projected/59b5b439-d6c7-403c-928a-3815511de21a-kube-api-access-r8cfd\") pod \"metrics-server-7cf9d5c9b8-zsvg2\" (UID: \"59b5b439-d6c7-403c-928a-3815511de21a\") " pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.769042 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.769007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:31:50.907280 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.907248 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-857644787b-vdjvp"] Sep 11 11:31:50.912031 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.912007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:50.919048 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.919019 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q"] Sep 11 11:31:50.925249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.925224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:50.925396 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.925267 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857644787b-vdjvp"] Sep 11 11:31:50.929307 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.929284 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-27zfv\"" Sep 11 11:31:50.929536 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.929519 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 11:31:50.940288 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:50.940228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q"] Sep 11 11:31:51.005289 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-service-ca\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.005488 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-trusted-ca-bundle\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.005488 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2d2q\" (UID: \"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.005488 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-oauth-serving-cert\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.005640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-oauth-config\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.005640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxrb\" (UniqueName: \"kubernetes.io/projected/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-kube-api-access-wpxrb\") pod \"monitoring-plugin-5d85778b9d-j2d2q\" (UID: \"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.005640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-serving-cert\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.005640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-config\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.005881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.005742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt2x\" (UniqueName: \"kubernetes.io/projected/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-kube-api-access-jkt2x\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.052788 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.052690 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2"] Sep 11 11:31:51.056915 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:51.056875 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b5b439_d6c7_403c_928a_3815511de21a.slice/crio-0a27695c95ffea4c3a54800c328f7f0875a9bee243d2385707236d6999f6ae6f WatchSource:0}: Error finding container 0a27695c95ffea4c3a54800c328f7f0875a9bee243d2385707236d6999f6ae6f: Status 404 returned error can't find the container with id 0a27695c95ffea4c3a54800c328f7f0875a9bee243d2385707236d6999f6ae6f Sep 11 11:31:51.107151 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-config\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.107359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt2x\" (UniqueName: \"kubernetes.io/projected/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-kube-api-access-jkt2x\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.107359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-service-ca\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.107359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-trusted-ca-bundle\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.107359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2d2q\" (UID: \"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.107567 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:51.107378 2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 11 11:31:51.107567 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:31:51.107461 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-monitoring-plugin-cert podName:a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a nodeName:}" failed. No retries permitted until 2026-09-11 11:31:51.607419948 +0000 UTC m=+88.143326947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-j2d2q" (UID: "a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a") : secret "monitoring-plugin-cert" not found Sep 11 11:31:51.107682 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-oauth-serving-cert\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.107777 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-oauth-config\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.107777 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107766 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxrb\" (UniqueName: \"kubernetes.io/projected/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-kube-api-access-wpxrb\") pod \"monitoring-plugin-5d85778b9d-j2d2q\" (UID: \"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.107891 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-serving-cert\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.108425 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.107952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-config\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.109277 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.109253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-oauth-serving-cert\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.110118 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.110092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-service-ca\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.110425 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.110400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-trusted-ca-bundle\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.111053 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.111030 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-serving-cert\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.112333 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.112307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-oauth-config\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.117417 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.117395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt2x\" (UniqueName: \"kubernetes.io/projected/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-kube-api-access-jkt2x\") pod \"console-857644787b-vdjvp\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.119003 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.118958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxrb\" (UniqueName: \"kubernetes.io/projected/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-kube-api-access-wpxrb\") pod \"monitoring-plugin-5d85778b9d-j2d2q\" (UID: \"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.242145 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.242111 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:31:51.372700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.372607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" event={"ID":"59b5b439-d6c7-403c-928a-3815511de21a","Type":"ContainerStarted","Data":"0a27695c95ffea4c3a54800c328f7f0875a9bee243d2385707236d6999f6ae6f"} Sep 11 11:31:51.375579 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.375549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerStarted","Data":"ea745cf05c6d76bf34fd8f76518d250148c47e315a7983a1b2ec6e713d0691b4"} Sep 11 11:31:51.378213 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.378186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" event={"ID":"3f2a1a95-d84c-42e0-8a94-a0571411210b","Type":"ContainerStarted","Data":"b5cd975c48ff90afd44d92993f145a69791ef900f35bd4a250b59c2f93e1d239"} Sep 11 11:31:51.383465 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.381708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-mhwfg" event={"ID":"0f79254d-a7c1-40ce-a08c-678276ff68ac","Type":"ContainerStarted","Data":"8e9d7b913deda9ef66db18fec28d27f5fb749dc2a10b306ed573ec26fb784ec0"} Sep 11 11:31:51.383465 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.383405 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-mhwfg" Sep 11 11:31:51.386381 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.386359 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" event={"ID":"ecefa57a-457c-478a-9b03-c40e9537873d","Type":"ContainerStarted","Data":"a4f5ca6846a765734385df1b8cf9b805b78613311e8aa556d3ec7a2e76941cdf"} Sep 11 11:31:51.386687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.386670 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:51.388838 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.388791 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" Sep 11 11:31:51.395823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.395777 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-mhwfg" Sep 11 11:31:51.403576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.402643 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-mhwfg" podStartSLOduration=1.701572494 podStartE2EDuration="21.402624909s" totalImagesPullingTime="19.701052415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:51.401766282 +0000 UTC m=+87.937673282" watchObservedRunningTime="2026-09-11 11:31:51.402624909 +0000 UTC m=+87.938531906" Sep 11 11:31:51.422516 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.421065 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b5d878674-jsl6r" podStartSLOduration=2.66563186 podStartE2EDuration="22.421045999s" totalImagesPullingTime="19.755414139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:51.420301654 +0000 UTC m=+87.956208654" watchObservedRunningTime="2026-09-11 11:31:51.421045999 +0000 UTC m=+87.956952997" Sep 11 11:31:51.561099 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.561063 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857644787b-vdjvp"] Sep 11 11:31:51.616359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.616318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2d2q\" (UID: \"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.622094 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.622061 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j2d2q\" (UID: \"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.849374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:51.849340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:51.881622 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:51.881582 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97671eb8_4ed0_4c9d_8579_66f8a49e4c76.slice/crio-7e2f8803d76115452adeb1ad538af04b7d6720a12a5c2914ac6e4199bb3f1ab1 WatchSource:0}: Error finding container 7e2f8803d76115452adeb1ad538af04b7d6720a12a5c2914ac6e4199bb3f1ab1: Status 404 returned error can't find the container with id 7e2f8803d76115452adeb1ad538af04b7d6720a12a5c2914ac6e4199bb3f1ab1 Sep 11 11:31:52.392865 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:52.392820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857644787b-vdjvp" event={"ID":"97671eb8-4ed0-4c9d-8579-66f8a49e4c76","Type":"ContainerStarted","Data":"7e2f8803d76115452adeb1ad538af04b7d6720a12a5c2914ac6e4199bb3f1ab1"} Sep 11 11:31:52.930056 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:52.929953 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f5dcd6d57-m697t"] Sep 11 11:31:52.937662 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:52.937633 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:31:53.624125 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:53.624052 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q"] Sep 11 11:31:53.696817 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:31:53.696788 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d26acc_7d8f_4bd4_88c2_7f9f39eff74a.slice/crio-f048950ed2fd3f3e5dfa1c3812d3a94c868d7b1b8bb7c3e612d692aa3fb7d84e WatchSource:0}: Error finding container f048950ed2fd3f3e5dfa1c3812d3a94c868d7b1b8bb7c3e612d692aa3fb7d84e: Status 404 returned error can't find the container with id f048950ed2fd3f3e5dfa1c3812d3a94c868d7b1b8bb7c3e612d692aa3fb7d84e Sep 11 11:31:54.423483 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.420241 2579 generic.go:356] "Generic (PLEG): container finished" podID="4c4f2da0-ee51-463b-8913-b63d884aeec0" containerID="3c6e5a4142815e84ba32fa96c24f7023f27fd32deffb7e63076268c6cf1ebdd0" exitCode=0 Sep 11 11:31:54.423483 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.420362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2qz5" event={"ID":"4c4f2da0-ee51-463b-8913-b63d884aeec0","Type":"ContainerDied","Data":"3c6e5a4142815e84ba32fa96c24f7023f27fd32deffb7e63076268c6cf1ebdd0"} Sep 11 11:31:54.425910 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.425872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" event={"ID":"3f2a1a95-d84c-42e0-8a94-a0571411210b","Type":"ContainerStarted","Data":"1aedbb00c4faa8d2215c5789344f0eb8b112c7ecaae0f7d48eaa54a478ff4dda"} Sep 11 11:31:54.427730 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.427682 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerID="bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4" exitCode=0 Sep 11 11:31:54.427837 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.427779 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4"} Sep 11 11:31:54.432649 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.432619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857644787b-vdjvp" event={"ID":"97671eb8-4ed0-4c9d-8579-66f8a49e4c76","Type":"ContainerStarted","Data":"51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1"} Sep 11 11:31:54.434542 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.434520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" event={"ID":"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a","Type":"ContainerStarted","Data":"f048950ed2fd3f3e5dfa1c3812d3a94c868d7b1b8bb7c3e612d692aa3fb7d84e"} Sep 11 11:31:54.488577 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:54.488529 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-857644787b-vdjvp" podStartSLOduration=4.488508322 podStartE2EDuration="4.488508322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:54.48618994 +0000 UTC m=+91.022096973" watchObservedRunningTime="2026-09-11 11:31:54.488508322 +0000 UTC m=+91.024415322" Sep 11 11:31:55.454238 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.454201 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2qz5" event={"ID":"4c4f2da0-ee51-463b-8913-b63d884aeec0","Type":"ContainerStarted","Data":"681c0060cb53cb0bfdd1bc9f2641df9a1bbb46184803dc1d4d9d6639cea7d552"} Sep 11 11:31:55.454759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.454269 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2qz5" event={"ID":"4c4f2da0-ee51-463b-8913-b63d884aeec0","Type":"ContainerStarted","Data":"c4616aa45757cf1655df1b5109c0e3f7c0822d83faddd5e9292f0808ba426fa3"} Sep 11 11:31:55.459570 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.458330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" event={"ID":"3f2a1a95-d84c-42e0-8a94-a0571411210b","Type":"ContainerStarted","Data":"8bab859bec1dcf5fc58e5f1e773d8c93768b5b0a08ee0768cf3179f2c308a7cd"} Sep 11 11:31:55.459570 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.458389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" event={"ID":"3f2a1a95-d84c-42e0-8a94-a0571411210b","Type":"ContainerStarted","Data":"0d32517cd7a43bb62cf1361068b9981b676043f26de482c0452681cedb3b5aa7"} Sep 11 11:31:55.465093 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.464091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" event={"ID":"59b5b439-d6c7-403c-928a-3815511de21a","Type":"ContainerStarted","Data":"66b28bb526a7d2871235e1f9f67731f8e5974c1474c716a56c446c7cd47a659f"} Sep 11 11:31:55.481425 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.479546 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n2qz5" podStartSLOduration=6.334634758 podStartE2EDuration="9.479526619s" totalImagesPullingTime="3.050307645s" totalInitContainerRuntime="94.584216ms" isStatefulPod=true podCreationTimestamp="2026-09-11 11:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:55.476668382 +0000 UTC m=+92.012575392" watchObservedRunningTime="2026-09-11 11:31:55.479526619 +0000 UTC m=+92.015433618" Sep 11 11:31:55.498705 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.498653 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-8js97" podStartSLOduration=5.8466606169999995 podStartE2EDuration="9.498633876s" totalImagesPullingTime="3.651973259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:55.497898606 +0000 UTC m=+92.033805605" watchObservedRunningTime="2026-09-11 11:31:55.498633876 +0000 UTC m=+92.034540878" Sep 11 11:31:55.525059 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:55.525005 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" podStartSLOduration=1.9019813509999999 podStartE2EDuration="5.52498534s" totalImagesPullingTime="3.623003989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:55.522869247 +0000 UTC m=+92.058776247" watchObservedRunningTime="2026-09-11 11:31:55.52498534 +0000 UTC m=+92.060892341" Sep 11 11:31:56.082313 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:56.082284 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857644787b-vdjvp"] Sep 11 11:31:57.476464 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:57.476417 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" event={"ID":"a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a","Type":"ContainerStarted","Data":"296253763ffe469eee0577d2c2d82e576439a5304029a941b0b60919b9464213"} Sep 11 11:31:57.476943 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:57.476534 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:57.479487 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:57.479454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerStarted","Data":"46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e"} Sep 11 11:31:57.479621 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:57.479524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerStarted","Data":"bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7"} Sep 11 11:31:57.482814 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:57.482788 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" Sep 11 11:31:57.498018 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:57.497034 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j2d2q" podStartSLOduration=4.156247139 podStartE2EDuration="7.497015821s" totalImagesPullingTime="3.340768682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:57.495949523 +0000 UTC m=+94.031856534" watchObservedRunningTime="2026-09-11 11:31:57.497015821 +0000 UTC m=+94.032922817" Sep 11 11:31:58.491673 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:58.491630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerStarted","Data":"e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd"} Sep 11 11:31:58.492155 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:58.491765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerStarted","Data":"e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec"} Sep 11 11:31:58.492155 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:58.491815 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerStarted","Data":"93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8"} Sep 11 11:31:59.502904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:59.502872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerStarted","Data":"a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f"} Sep 11 11:31:59.534072 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:31:59.533994 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.637425735 podStartE2EDuration="12.533977945s" totalImagesPullingTime="6.882986061s" totalInitContainerRuntime="13.566149ms" isStatefulPod=false podCreationTimestamp="2026-09-11 11:31:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:31:59.532337017 +0000 UTC m=+96.068244015" watchObservedRunningTime="2026-09-11 11:31:59.533977945 +0000 UTC m=+96.069884944" Sep 11 11:32:01.242445 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:01.242408 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:32:07.063290 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:07.063250 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796789d579-gfgvw"] Sep 11 11:32:10.419009 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.418957 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c8cb75595-dddn4" podUID="df247472-9d6b-4bf4-a1bb-615ca1312714" containerName="console" containerID="cri-o://b9cea677f6d675bccea50c3b848c3ab1af3103df4f290ddf611b93767a9a08e3" gracePeriod=15 Sep 11 11:32:10.546106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.545965 2579 generic.go:356] "Generic (PLEG): container finished" podID="df247472-9d6b-4bf4-a1bb-615ca1312714" containerID="b9cea677f6d675bccea50c3b848c3ab1af3103df4f290ddf611b93767a9a08e3" exitCode=0 Sep 11 11:32:10.546106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.546034 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8cb75595-dddn4" event={"ID":"df247472-9d6b-4bf4-a1bb-615ca1312714","Type":"ContainerDied","Data":"b9cea677f6d675bccea50c3b848c3ab1af3103df4f290ddf611b93767a9a08e3"} Sep 11 11:32:10.679980 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.679901 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:32:10.692844 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.692822 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd4ch\" (UniqueName: \"kubernetes.io/projected/df247472-9d6b-4bf4-a1bb-615ca1312714-kube-api-access-jd4ch\") pod \"df247472-9d6b-4bf4-a1bb-615ca1312714\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " Sep 11 11:32:10.692965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.692860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-console-config\") pod \"df247472-9d6b-4bf4-a1bb-615ca1312714\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " Sep 11 11:32:10.692965 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.692888 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-oauth-serving-cert\") pod \"df247472-9d6b-4bf4-a1bb-615ca1312714\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " Sep 11 11:32:10.693082 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.693020 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-service-ca\") pod \"df247472-9d6b-4bf4-a1bb-615ca1312714\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " Sep 11 11:32:10.693082 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.693077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-oauth-config\") pod \"df247472-9d6b-4bf4-a1bb-615ca1312714\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " Sep 11 11:32:10.693189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.693123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-serving-cert\") pod \"df247472-9d6b-4bf4-a1bb-615ca1312714\" (UID: \"df247472-9d6b-4bf4-a1bb-615ca1312714\") " Sep 11 11:32:10.693359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.693326 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-console-config" (OuterVolumeSpecName: "console-config") pod "df247472-9d6b-4bf4-a1bb-615ca1312714" (UID: "df247472-9d6b-4bf4-a1bb-615ca1312714"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:10.693359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.693340 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "df247472-9d6b-4bf4-a1bb-615ca1312714" (UID: "df247472-9d6b-4bf4-a1bb-615ca1312714"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:10.693515 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.693423 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-service-ca" (OuterVolumeSpecName: "service-ca") pod "df247472-9d6b-4bf4-a1bb-615ca1312714" (UID: "df247472-9d6b-4bf4-a1bb-615ca1312714"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:10.695372 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.695337 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "df247472-9d6b-4bf4-a1bb-615ca1312714" (UID: "df247472-9d6b-4bf4-a1bb-615ca1312714"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:10.695471 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.695377 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df247472-9d6b-4bf4-a1bb-615ca1312714-kube-api-access-jd4ch" (OuterVolumeSpecName: "kube-api-access-jd4ch") pod "df247472-9d6b-4bf4-a1bb-615ca1312714" (UID: "df247472-9d6b-4bf4-a1bb-615ca1312714"). InnerVolumeSpecName "kube-api-access-jd4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:32:10.695471 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.695384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "df247472-9d6b-4bf4-a1bb-615ca1312714" (UID: "df247472-9d6b-4bf4-a1bb-615ca1312714"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:10.769832 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.769811 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:32:10.769931 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.769862 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:32:10.794693 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.794667 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-oauth-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:10.794693 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.794693 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df247472-9d6b-4bf4-a1bb-615ca1312714-console-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:10.794878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.794709 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd4ch\" (UniqueName: \"kubernetes.io/projected/df247472-9d6b-4bf4-a1bb-615ca1312714-kube-api-access-jd4ch\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:10.794878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.794832 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-console-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:10.794878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.794846 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-oauth-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:10.794878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:10.794862 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df247472-9d6b-4bf4-a1bb-615ca1312714-service-ca\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:11.331185 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:11.331156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-prvf7" Sep 11 11:32:11.551391 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:11.551367 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c8cb75595-dddn4" Sep 11 11:32:11.551881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:11.551414 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c8cb75595-dddn4" event={"ID":"df247472-9d6b-4bf4-a1bb-615ca1312714","Type":"ContainerDied","Data":"b819e88440509109e1e69c3a227deb9389dcea75b872960599470a20acd5315e"} Sep 11 11:32:11.551881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:11.551476 2579 scope.go:117] "RemoveContainer" containerID="b9cea677f6d675bccea50c3b848c3ab1af3103df4f290ddf611b93767a9a08e3" Sep 11 11:32:11.572025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:11.572006 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c8cb75595-dddn4"] Sep 11 11:32:11.575710 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:11.575690 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c8cb75595-dddn4"] Sep 11 11:32:11.899764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:11.899705 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df247472-9d6b-4bf4-a1bb-615ca1312714" path="/var/lib/kubelet/pods/df247472-9d6b-4bf4-a1bb-615ca1312714/volumes" Sep 11 11:32:17.959674 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:17.959629 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" podUID="9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" containerName="registry" containerID="cri-o://1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032" gracePeriod=30 Sep 11 11:32:18.238875 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.238794 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:32:18.250360 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250330 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-ca-trust-extracted\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.250499 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250365 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-image-registry-private-configuration\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.250499 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250418 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-bound-sa-token\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.250499 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250450 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.250499 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250480 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ckr7\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-kube-api-access-9ckr7\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.250739 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250535 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-trusted-ca\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.250739 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250560 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-certificates\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.250739 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.250609 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-installation-pull-secrets\") pod \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\" (UID: \"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933\") " Sep 11 11:32:18.251210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.251182 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:18.251459 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.251414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:18.253102 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.253073 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:32:18.253389 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.253361 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:18.253517 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.253495 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:18.253600 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.253535 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:32:18.253744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.253705 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-kube-api-access-9ckr7" (OuterVolumeSpecName: "kube-api-access-9ckr7") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "kube-api-access-9ckr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:32:18.262316 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.262288 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" (UID: "9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:32:18.352137 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352102 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.352137 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352131 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckr7\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-kube-api-access-9ckr7\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.352137 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352140 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-trusted-ca\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.352137 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352149 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-registry-certificates\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.352394 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352157 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-installation-pull-secrets\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.352394 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352168 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-ca-trust-extracted\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.352394 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352182 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-image-registry-private-configuration\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.352394 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.352190 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933-bound-sa-token\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:18.583173 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.583089 2579 generic.go:356] "Generic (PLEG): container finished" podID="9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" containerID="1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032" exitCode=0 Sep 11 11:32:18.583173 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.583131 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" Sep 11 11:32:18.583390 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.583183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" event={"ID":"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933","Type":"ContainerDied","Data":"1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032"} Sep 11 11:32:18.583390 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.583235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5dcd6d57-m697t" event={"ID":"9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933","Type":"ContainerDied","Data":"4e474c8d793af80bc9e3234a1a5c4ce1c94059e672571b049509bd328f543d42"} Sep 11 11:32:18.583390 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.583261 2579 scope.go:117] "RemoveContainer" containerID="1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032" Sep 11 11:32:18.596254 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.596232 2579 scope.go:117] "RemoveContainer" containerID="1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032" Sep 11 11:32:18.596508 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:32:18.596490 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032\": container with ID starting with 1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032 not found: ID does not exist" containerID="1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032" Sep 11 11:32:18.596562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.596515 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032"} err="failed to get container status \"1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032\": rpc error: code = NotFound desc = could not find container \"1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032\": container with ID starting with 1ab2f84280ca7cd4cec37c67dc1f893f801400b36a55aaf15813cbc283d08032 not found: ID does not exist" Sep 11 11:32:18.603584 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.603565 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f5dcd6d57-m697t"] Sep 11 11:32:18.610284 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:18.610264 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f5dcd6d57-m697t"] Sep 11 11:32:19.902334 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:19.902300 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" path="/var/lib/kubelet/pods/9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933/volumes" Sep 11 11:32:21.495062 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.495016 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-857644787b-vdjvp" podUID="97671eb8-4ed0-4c9d-8579-66f8a49e4c76" containerName="console" containerID="cri-o://51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1" gracePeriod=15 Sep 11 11:32:21.779919 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.779866 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:32:21.880114 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880088 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkt2x\" (UniqueName: \"kubernetes.io/projected/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-kube-api-access-jkt2x\") pod \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " Sep 11 11:32:21.880255 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880133 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-serving-cert\") pod \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " Sep 11 11:32:21.880255 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880150 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-trusted-ca-bundle\") pod \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " Sep 11 11:32:21.880255 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880171 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-oauth-config\") pod \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " Sep 11 11:32:21.880407 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880306 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-service-ca\") pod \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " Sep 11 11:32:21.880460 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880421 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-oauth-serving-cert\") pod \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " Sep 11 11:32:21.880514 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880474 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-config\") pod \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\" (UID: \"97671eb8-4ed0-4c9d-8579-66f8a49e4c76\") " Sep 11 11:32:21.880674 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880648 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97671eb8-4ed0-4c9d-8579-66f8a49e4c76" (UID: "97671eb8-4ed0-4c9d-8579-66f8a49e4c76"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:21.880757 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880694 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-service-ca" (OuterVolumeSpecName: "service-ca") pod "97671eb8-4ed0-4c9d-8579-66f8a49e4c76" (UID: "97671eb8-4ed0-4c9d-8579-66f8a49e4c76"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:21.880811 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880790 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97671eb8-4ed0-4c9d-8579-66f8a49e4c76" (UID: "97671eb8-4ed0-4c9d-8579-66f8a49e4c76"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:21.880811 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.880799 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-config" (OuterVolumeSpecName: "console-config") pod "97671eb8-4ed0-4c9d-8579-66f8a49e4c76" (UID: "97671eb8-4ed0-4c9d-8579-66f8a49e4c76"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:21.882371 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.882348 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97671eb8-4ed0-4c9d-8579-66f8a49e4c76" (UID: "97671eb8-4ed0-4c9d-8579-66f8a49e4c76"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:21.882429 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.882412 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97671eb8-4ed0-4c9d-8579-66f8a49e4c76" (UID: "97671eb8-4ed0-4c9d-8579-66f8a49e4c76"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:21.882580 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.882565 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-kube-api-access-jkt2x" (OuterVolumeSpecName: "kube-api-access-jkt2x") pod "97671eb8-4ed0-4c9d-8579-66f8a49e4c76" (UID: "97671eb8-4ed0-4c9d-8579-66f8a49e4c76"). InnerVolumeSpecName "kube-api-access-jkt2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:32:21.981501 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.981477 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkt2x\" (UniqueName: \"kubernetes.io/projected/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-kube-api-access-jkt2x\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:21.981501 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.981499 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:21.981629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.981507 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-trusted-ca-bundle\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:21.981629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.981515 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-oauth-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:21.981629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.981523 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-service-ca\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:21.981629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.981531 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-oauth-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:21.981629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:21.981541 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97671eb8-4ed0-4c9d-8579-66f8a49e4c76-console-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:22.600025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.599990 2579 generic.go:356] "Generic (PLEG): container finished" podID="97671eb8-4ed0-4c9d-8579-66f8a49e4c76" containerID="51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1" exitCode=0 Sep 11 11:32:22.600488 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.600038 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857644787b-vdjvp" Sep 11 11:32:22.600488 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.600042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857644787b-vdjvp" event={"ID":"97671eb8-4ed0-4c9d-8579-66f8a49e4c76","Type":"ContainerDied","Data":"51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1"} Sep 11 11:32:22.600488 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.600167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857644787b-vdjvp" event={"ID":"97671eb8-4ed0-4c9d-8579-66f8a49e4c76","Type":"ContainerDied","Data":"7e2f8803d76115452adeb1ad538af04b7d6720a12a5c2914ac6e4199bb3f1ab1"} Sep 11 11:32:22.600488 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.600192 2579 scope.go:117] "RemoveContainer" containerID="51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1" Sep 11 11:32:22.608596 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.608582 2579 scope.go:117] "RemoveContainer" containerID="51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1" Sep 11 11:32:22.608871 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:32:22.608854 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1\": container with ID starting with 51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1 not found: ID does not exist" containerID="51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1" Sep 11 11:32:22.608939 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.608880 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1"} err="failed to get container status \"51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1\": rpc error: code = NotFound desc = could not find container \"51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1\": container with ID starting with 51ec0a8d69e91bee99e205ae9dd1af7c89bb445e4e7eb34a2bc029e4ba110be1 not found: ID does not exist" Sep 11 11:32:22.617669 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.617642 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857644787b-vdjvp"] Sep 11 11:32:22.621174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:22.621153 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-857644787b-vdjvp"] Sep 11 11:32:23.900428 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:23.900392 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97671eb8-4ed0-4c9d-8579-66f8a49e4c76" path="/var/lib/kubelet/pods/97671eb8-4ed0-4c9d-8579-66f8a49e4c76/volumes" Sep 11 11:32:30.779423 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:30.779037 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:32:30.786955 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:30.786934 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cf9d5c9b8-zsvg2" Sep 11 11:32:32.082898 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.082825 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-796789d579-gfgvw" podUID="e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" containerName="console" containerID="cri-o://a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37" gracePeriod=15 Sep 11 11:32:32.387710 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.387684 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:32:32.463164 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-config\") pod \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " Sep 11 11:32:32.463353 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463187 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-service-ca\") pod \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " Sep 11 11:32:32.463353 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463216 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htn9w\" (UniqueName: \"kubernetes.io/projected/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-kube-api-access-htn9w\") pod \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " Sep 11 11:32:32.463456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-serving-cert\") pod \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " Sep 11 11:32:32.463512 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-trusted-ca-bundle\") pod \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " Sep 11 11:32:32.463512 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463490 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-config" (OuterVolumeSpecName: "console-config") pod "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" (UID: "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:32.463512 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-oauth-serving-cert\") pod \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " Sep 11 11:32:32.463664 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463538 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-service-ca" (OuterVolumeSpecName: "service-ca") pod "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" (UID: "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:32.463664 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463591 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-oauth-config\") pod \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\" (UID: \"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd\") " Sep 11 11:32:32.463821 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463765 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" (UID: "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:32.463881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463837 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:32.463881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463853 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-service-ca\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:32.463881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463865 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-trusted-ca-bundle\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:32.464037 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.463912 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" (UID: "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:32:32.465526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.465499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-kube-api-access-htn9w" (OuterVolumeSpecName: "kube-api-access-htn9w") pod "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" (UID: "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd"). InnerVolumeSpecName "kube-api-access-htn9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:32:32.465960 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.465937 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" (UID: "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:32.466195 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.466176 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" (UID: "e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:32:32.565002 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.564966 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-oauth-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:32.565002 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.565000 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htn9w\" (UniqueName: \"kubernetes.io/projected/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-kube-api-access-htn9w\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:32.565002 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.565011 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-console-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:32.565210 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.565019 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd-oauth-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:32:32.644077 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.643557 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" containerID="a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37" exitCode=0 Sep 11 11:32:32.644077 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.643607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796789d579-gfgvw" event={"ID":"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd","Type":"ContainerDied","Data":"a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37"} Sep 11 11:32:32.644077 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.643657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-796789d579-gfgvw" event={"ID":"e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd","Type":"ContainerDied","Data":"ea525c48f39c66a0ef9dd3689cfd66997f98351e986c8d39af0381e4fb6c5dce"} Sep 11 11:32:32.644077 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.643689 2579 scope.go:117] "RemoveContainer" containerID="a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37" Sep 11 11:32:32.644077 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.643895 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-796789d579-gfgvw" Sep 11 11:32:32.661875 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.661857 2579 scope.go:117] "RemoveContainer" containerID="a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37" Sep 11 11:32:32.662149 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:32:32.662126 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37\": container with ID starting with a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37 not found: ID does not exist" containerID="a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37" Sep 11 11:32:32.662231 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.662161 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37"} err="failed to get container status \"a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37\": rpc error: code = NotFound desc = could not find container \"a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37\": container with ID starting with a9e54259ea4bef9ad74c0a2f8ad566bb248e423518c3aec037373ef8f9ec3e37 not found: ID does not exist" Sep 11 11:32:32.674007 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.673980 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-796789d579-gfgvw"] Sep 11 11:32:32.677508 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:32.677482 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-796789d579-gfgvw"] Sep 11 11:32:33.899904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:32:33.899870 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" path="/var/lib/kubelet/pods/e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd/volumes" Sep 11 11:33:06.469699 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.469663 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:33:06.470189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.469962 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="alertmanager" containerID="cri-o://bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7" gracePeriod=120 Sep 11 11:33:06.470189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.470036 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-metric" containerID="cri-o://e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd" gracePeriod=120 Sep 11 11:33:06.470189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.470060 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-web" containerID="cri-o://93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8" gracePeriod=120 Sep 11 11:33:06.470189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.470099 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="prom-label-proxy" containerID="cri-o://a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f" gracePeriod=120 Sep 11 11:33:06.470189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.470153 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy" containerID="cri-o://e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec" gracePeriod=120 Sep 11 11:33:06.470189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.470101 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="config-reloader" containerID="cri-o://46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e" gracePeriod=120 Sep 11 11:33:06.775926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775839 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerID="a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f" exitCode=0 Sep 11 11:33:06.775926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775867 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerID="e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd" exitCode=0 Sep 11 11:33:06.775926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775876 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerID="e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec" exitCode=0 Sep 11 11:33:06.775926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775883 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerID="46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e" exitCode=0 Sep 11 11:33:06.775926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775888 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerID="bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7" exitCode=0 Sep 11 11:33:06.775926 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775907 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f"} Sep 11 11:33:06.776216 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775970 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd"} Sep 11 11:33:06.776216 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.775996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec"} Sep 11 11:33:06.776216 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.776021 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e"} Sep 11 11:33:06.776216 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:06.776044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7"} Sep 11 11:33:07.704923 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.704901 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:07.783756 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.783704 2579 generic.go:356] "Generic (PLEG): container finished" podID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerID="93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8" exitCode=0 Sep 11 11:33:07.783904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.783768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8"} Sep 11 11:33:07.783904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.783818 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:07.783904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.783826 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b56095f1-12d4-41e4-bbc0-8f3134678f6e","Type":"ContainerDied","Data":"ea745cf05c6d76bf34fd8f76518d250148c47e315a7983a1b2ec6e713d0691b4"} Sep 11 11:33:07.783904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.783861 2579 scope.go:117] "RemoveContainer" containerID="a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f" Sep 11 11:33:07.791509 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.791491 2579 scope.go:117] "RemoveContainer" containerID="e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd" Sep 11 11:33:07.798314 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.798298 2579 scope.go:117] "RemoveContainer" containerID="e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec" Sep 11 11:33:07.806866 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.806848 2579 scope.go:117] "RemoveContainer" containerID="93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8" Sep 11 11:33:07.813376 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.813359 2579 scope.go:117] "RemoveContainer" containerID="46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e" Sep 11 11:33:07.819948 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.819932 2579 scope.go:117] "RemoveContainer" containerID="bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7" Sep 11 11:33:07.823959 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.823942 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-kube-api-access-p7bdl\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824027 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.823982 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824027 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824006 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-tls-assets\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824129 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824107 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824183 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824160 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-main-db\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824231 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824196 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824282 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824228 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-cluster-tls-config\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824282 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824260 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-web-config\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824380 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824316 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-metrics-client-ca\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.824431 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824383 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:33:07.824666 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:33:07.824766 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824668 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:33:07.824766 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824760 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-volume\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.825044 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824810 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.825044 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.824853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-out\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.825151 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.825031 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls\") pod \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\" (UID: \"b56095f1-12d4-41e4-bbc0-8f3134678f6e\") " Sep 11 11:33:07.825485 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.825301 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-metrics-client-ca\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.825485 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.825326 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.825485 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.825342 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-alertmanager-main-db\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.827022 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.826995 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:33:07.828249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.827958 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-kube-api-access-p7bdl" (OuterVolumeSpecName: "kube-api-access-p7bdl") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "kube-api-access-p7bdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:33:07.828249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.828170 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:33:07.828777 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.828734 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-out" (OuterVolumeSpecName: "config-out") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:33:07.828777 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.828761 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:33:07.829162 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.829055 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:33:07.829230 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.829173 2579 scope.go:117] "RemoveContainer" containerID="bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4" Sep 11 11:33:07.829459 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.829373 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:33:07.829975 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.829950 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-volume" (OuterVolumeSpecName: "config-volume") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:33:07.832392 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.832369 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:33:07.838479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.838431 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-web-config" (OuterVolumeSpecName: "web-config") pod "b56095f1-12d4-41e4-bbc0-8f3134678f6e" (UID: "b56095f1-12d4-41e4-bbc0-8f3134678f6e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:33:07.844474 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.844455 2579 scope.go:117] "RemoveContainer" containerID="a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f" Sep 11 11:33:07.844764 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:33:07.844746 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f\": container with ID starting with a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f not found: ID does not exist" containerID="a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f" Sep 11 11:33:07.844826 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.844773 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f"} err="failed to get container status \"a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f\": rpc error: code = NotFound desc = could not find container \"a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f\": container with ID starting with a28dc64d66c50b17329ed317b4ab0e9358efa3ffaeb4686ad732cbb75da6f08f not found: ID does not exist" Sep 11 11:33:07.844826 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.844790 2579 scope.go:117] "RemoveContainer" containerID="e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd" Sep 11 11:33:07.845069 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:33:07.845054 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd\": container with ID starting with e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd not found: ID does not exist" containerID="e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd" Sep 11 11:33:07.845113 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845074 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd"} err="failed to get container status \"e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd\": rpc error: code = NotFound desc = could not find container \"e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd\": container with ID starting with e412dcfc374f2ab5473d15467dae8e2dd4e93646babf41a52580b9f1f0636cfd not found: ID does not exist" Sep 11 11:33:07.845113 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845089 2579 scope.go:117] "RemoveContainer" containerID="e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec" Sep 11 11:33:07.845319 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:33:07.845303 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec\": container with ID starting with e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec not found: ID does not exist" containerID="e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec" Sep 11 11:33:07.845359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845325 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec"} err="failed to get container status \"e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec\": rpc error: code = NotFound desc = could not find container \"e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec\": container with ID starting with e916758a8997fc16880989ff6bfa01ab0b85243dc5dd20187101171d4fd9ffec not found: ID does not exist" Sep 11 11:33:07.845359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845338 2579 scope.go:117] "RemoveContainer" containerID="93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8" Sep 11 11:33:07.845573 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:33:07.845556 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8\": container with ID starting with 93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8 not found: ID does not exist" containerID="93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8" Sep 11 11:33:07.845613 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845576 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8"} err="failed to get container status \"93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8\": rpc error: code = NotFound desc = could not find container \"93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8\": container with ID starting with 93a3e644f22ccd3ee3196c0250fc6dee9f24b9acfe030ca0300be34f2bcf0de8 not found: ID does not exist" Sep 11 11:33:07.845613 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845589 2579 scope.go:117] "RemoveContainer" containerID="46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e" Sep 11 11:33:07.845829 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:33:07.845812 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e\": container with ID starting with 46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e not found: ID does not exist" containerID="46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e" Sep 11 11:33:07.845880 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845831 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e"} err="failed to get container status \"46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e\": rpc error: code = NotFound desc = could not find container \"46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e\": container with ID starting with 46f52502c6aef91d359450c40fba302463fd096e77399fbb2541bd6ed7c3e55e not found: ID does not exist" Sep 11 11:33:07.845880 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.845844 2579 scope.go:117] "RemoveContainer" containerID="bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7" Sep 11 11:33:07.846092 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:33:07.846075 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7\": container with ID starting with bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7 not found: ID does not exist" containerID="bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7" Sep 11 11:33:07.846148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.846094 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7"} err="failed to get container status \"bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7\": rpc error: code = NotFound desc = could not find container \"bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7\": container with ID starting with bf5bd91437c8d8f4ae2e91c7c721159c4934e5eddd01fe6634c27f8c8ceaa0a7 not found: ID does not exist" Sep 11 11:33:07.846148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.846107 2579 scope.go:117] "RemoveContainer" containerID="bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4" Sep 11 11:33:07.846359 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:33:07.846342 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4\": container with ID starting with bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4 not found: ID does not exist" containerID="bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4" Sep 11 11:33:07.846397 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.846362 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4"} err="failed to get container status \"bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4\": rpc error: code = NotFound desc = could not find container \"bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4\": container with ID starting with bf083bec4165f211a615770a19e808c6a39007206ad2531cffa42bad7825d0e4 not found: ID does not exist" Sep 11 11:33:07.925914 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925892 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.925914 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925913 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-cluster-tls-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925942 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-web-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925951 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-volume\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925959 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925967 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b56095f1-12d4-41e4-bbc0-8f3134678f6e-config-out\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925975 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-main-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925983 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7bdl\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-kube-api-access-p7bdl\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925990 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b56095f1-12d4-41e4-bbc0-8f3134678f6e-tls-assets\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:07.926025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:07.925999 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b56095f1-12d4-41e4-bbc0-8f3134678f6e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:33:08.101748 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.101708 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:33:08.106690 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.106670 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:33:08.129999 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.129977 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:33:08.130226 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130212 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="config-reloader" Sep 11 11:33:08.130267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130229 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="config-reloader" Sep 11 11:33:08.130267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130242 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="prom-label-proxy" Sep 11 11:33:08.130267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130248 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="prom-label-proxy" Sep 11 11:33:08.130267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130255 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" containerName="registry" Sep 11 11:33:08.130267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130260 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" containerName="registry" Sep 11 11:33:08.130267 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130266 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97671eb8-4ed0-4c9d-8579-66f8a49e4c76" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130271 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97671eb8-4ed0-4c9d-8579-66f8a49e4c76" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130277 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df247472-9d6b-4bf4-a1bb-615ca1312714" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130282 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df247472-9d6b-4bf4-a1bb-615ca1312714" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130289 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-web" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130297 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-web" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130306 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130311 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130318 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="alertmanager" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130324 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="alertmanager" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130332 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-metric" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130337 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-metric" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130342 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="init-config-reloader" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130347 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="init-config-reloader" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130354 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130359 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130404 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="alertmanager" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130411 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="config-reloader" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130417 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e4061b8-cd8f-4c1c-8b97-dcf08b1bc933" containerName="registry" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130421 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df247472-9d6b-4bf4-a1bb-615ca1312714" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130426 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="prom-label-proxy" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130431 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-metric" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130436 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f5dcd7-0c1c-4994-b0be-837d04d1c1bd" containerName="console" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130441 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy-web" Sep 11 11:33:08.130436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130447 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97671eb8-4ed0-4c9d-8579-66f8a49e4c76" containerName="console" Sep 11 11:33:08.131165 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.130453 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" containerName="kube-rbac-proxy" Sep 11 11:33:08.134123 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.134108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.136708 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.136684 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 11:33:08.136835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.136790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 11:33:08.136987 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.136969 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 11:33:08.137273 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.137258 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 11:33:08.137333 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.137324 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 11:33:08.137457 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.137441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 11:33:08.137516 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.137443 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q24g9\"" Sep 11 11:33:08.137516 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.137505 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 11:33:08.137852 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.137832 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 11:33:08.142699 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.142680 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 11:33:08.147392 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.147372 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:33:08.228262 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6aa8c45-ce35-499a-8514-3c8265a9472b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228363 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228363 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228449 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6aa8c45-ce35-499a-8514-3c8265a9472b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228449 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228449 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6aa8c45-ce35-499a-8514-3c8265a9472b-config-out\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228564 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228564 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-web-config\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228564 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228564 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8c45-ce35-499a-8514-3c8265a9472b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228564 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6aa8c45-ce35-499a-8514-3c8265a9472b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228708 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z866\" (UniqueName: \"kubernetes.io/projected/f6aa8c45-ce35-499a-8514-3c8265a9472b-kube-api-access-6z866\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.228708 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.228648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.329900 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.329880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.329957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.329907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6aa8c45-ce35-499a-8514-3c8265a9472b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.329957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.329926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.329957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.329953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6aa8c45-ce35-499a-8514-3c8265a9472b-config-out\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330096 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.329971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330096 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-web-config\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330096 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330096 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8c45-ce35-499a-8514-3c8265a9472b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330277 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6aa8c45-ce35-499a-8514-3c8265a9472b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330277 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z866\" (UniqueName: \"kubernetes.io/projected/f6aa8c45-ce35-499a-8514-3c8265a9472b-kube-api-access-6z866\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330277 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330277 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6aa8c45-ce35-499a-8514-3c8265a9472b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.330277 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.330185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.331336 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.331287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6aa8c45-ce35-499a-8514-3c8265a9472b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.331539 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.331521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aa8c45-ce35-499a-8514-3c8265a9472b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333118 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333118 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f6aa8c45-ce35-499a-8514-3c8265a9472b-config-out\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333258 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333342 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f6aa8c45-ce35-499a-8514-3c8265a9472b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333397 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f6aa8c45-ce35-499a-8514-3c8265a9472b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333449 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-web-config\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333503 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.333881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.333862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.334354 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.334335 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-config-volume\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.335057 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.335042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f6aa8c45-ce35-499a-8514-3c8265a9472b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.338987 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.338968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z866\" (UniqueName: \"kubernetes.io/projected/f6aa8c45-ce35-499a-8514-3c8265a9472b-kube-api-access-6z866\") pod \"alertmanager-main-0\" (UID: \"f6aa8c45-ce35-499a-8514-3c8265a9472b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.443243 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.443154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 11:33:08.715116 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.715089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 11:33:08.717650 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:33:08.717623 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aa8c45_ce35_499a_8514_3c8265a9472b.slice/crio-7660982705a3a528e58f1eab770852ec4caf1a8f747a175974a1cf7805706dc3 WatchSource:0}: Error finding container 7660982705a3a528e58f1eab770852ec4caf1a8f747a175974a1cf7805706dc3: Status 404 returned error can't find the container with id 7660982705a3a528e58f1eab770852ec4caf1a8f747a175974a1cf7805706dc3 Sep 11 11:33:08.789885 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.789847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"dc5cbb9fdf821d1f4e1bb0a25d6158cdd102ea384a4c7ed4e9a4cdc27254a669"} Sep 11 11:33:08.790003 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:08.789990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"7660982705a3a528e58f1eab770852ec4caf1a8f747a175974a1cf7805706dc3"} Sep 11 11:33:09.795141 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:09.795107 2579 generic.go:356] "Generic (PLEG): container finished" podID="f6aa8c45-ce35-499a-8514-3c8265a9472b" containerID="dc5cbb9fdf821d1f4e1bb0a25d6158cdd102ea384a4c7ed4e9a4cdc27254a669" exitCode=0 Sep 11 11:33:09.795504 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:09.795172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerDied","Data":"dc5cbb9fdf821d1f4e1bb0a25d6158cdd102ea384a4c7ed4e9a4cdc27254a669"} Sep 11 11:33:09.900819 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:09.900793 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56095f1-12d4-41e4-bbc0-8f3134678f6e" path="/var/lib/kubelet/pods/b56095f1-12d4-41e4-bbc0-8f3134678f6e/volumes" Sep 11 11:33:10.802015 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:10.801983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"d4cc8af647ff6eb96fd2bb08ea7c629f8a502805b27265866280c9ab3ca9e6b3"} Sep 11 11:33:10.802397 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:10.802044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"e18c3c82aab030bc0d7bd7c64536fc0b967b2a6c295e1563dba0f1012f1323eb"} Sep 11 11:33:10.802397 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:10.802071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"2abab9da6fd781f9e95013342919dc5aa740568d1a1576710aac6ce3372f9a35"} Sep 11 11:33:10.802397 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:10.802093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"1d483ee19419fee2207860c0f939115ae34c8095d6e94d666513a9a9804d19fa"} Sep 11 11:33:10.802397 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:10.802116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"736a3d4b8f974261780851bbb2caec32e269871976344199f75c966ab3e1ff00"} Sep 11 11:33:10.802397 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:10.802136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f6aa8c45-ce35-499a-8514-3c8265a9472b","Type":"ContainerStarted","Data":"74de15368044f421595b790fcd756c87c73ca940756b2293330b64959191718a"} Sep 11 11:33:10.833870 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:10.833820 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8163669000000002 podStartE2EDuration="2.833806262s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.439362ms" isStatefulPod=false podCreationTimestamp="2026-09-11 11:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:33:10.828196684 +0000 UTC m=+167.364103681" watchObservedRunningTime="2026-09-11 11:33:10.833806262 +0000 UTC m=+167.369713255" Sep 11 11:33:14.889600 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.889562 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59cd4c9dd6-mvwtb"] Sep 11 11:33:14.891786 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.891768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:14.895879 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.895851 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 11:33:14.895879 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.895866 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 11:33:14.896088 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.895859 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 11:33:14.896088 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.895861 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 11:33:14.896088 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.895856 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cs8hc\"" Sep 11 11:33:14.896088 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.895965 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 11:33:14.900648 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.900629 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 11:33:14.905059 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:14.905039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59cd4c9dd6-mvwtb"] Sep 11 11:33:15.079252 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.079219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-trusted-ca-bundle\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.079443 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.079280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgq8\" (UniqueName: \"kubernetes.io/projected/e302366f-e71d-4a9b-aa89-d353529a0dcc-kube-api-access-qqgq8\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.079443 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.079332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-serving-cert\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.079443 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.079354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-oauth-config\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.079443 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.079375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-oauth-serving-cert\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.079443 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.079425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-service-ca\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.079634 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.079474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-config\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.181241 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.181138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-serving-cert\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.181241 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.181209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-oauth-config\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.181458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.181244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-oauth-serving-cert\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.181458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.181277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-service-ca\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.181458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.181327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-config\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.181458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.181383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-trusted-ca-bundle\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.181458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.181431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgq8\" (UniqueName: \"kubernetes.io/projected/e302366f-e71d-4a9b-aa89-d353529a0dcc-kube-api-access-qqgq8\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.183321 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.183288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-oauth-serving-cert\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.183905 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.183881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-service-ca\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.186201 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.184426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-config\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.186201 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.185431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-trusted-ca-bundle\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.186621 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.186591 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-oauth-config\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.188037 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.188019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-serving-cert\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.190089 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.190067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgq8\" (UniqueName: \"kubernetes.io/projected/e302366f-e71d-4a9b-aa89-d353529a0dcc-kube-api-access-qqgq8\") pod \"console-59cd4c9dd6-mvwtb\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.200928 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.200905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:15.468604 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.468574 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59cd4c9dd6-mvwtb"] Sep 11 11:33:15.471243 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:33:15.471216 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode302366f_e71d_4a9b_aa89_d353529a0dcc.slice/crio-947ffe157303ab4b0834ea0575bd03592e9722308bf746b2230a51f571a5491e WatchSource:0}: Error finding container 947ffe157303ab4b0834ea0575bd03592e9722308bf746b2230a51f571a5491e: Status 404 returned error can't find the container with id 947ffe157303ab4b0834ea0575bd03592e9722308bf746b2230a51f571a5491e Sep 11 11:33:15.823073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.822982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cd4c9dd6-mvwtb" event={"ID":"e302366f-e71d-4a9b-aa89-d353529a0dcc","Type":"ContainerStarted","Data":"2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263"} Sep 11 11:33:15.823073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.823029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cd4c9dd6-mvwtb" event={"ID":"e302366f-e71d-4a9b-aa89-d353529a0dcc","Type":"ContainerStarted","Data":"947ffe157303ab4b0834ea0575bd03592e9722308bf746b2230a51f571a5491e"} Sep 11 11:33:15.841234 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:15.841185 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59cd4c9dd6-mvwtb" podStartSLOduration=1.841167758 podStartE2EDuration="1.841167758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:33:15.839930956 +0000 UTC m=+172.375837956" watchObservedRunningTime="2026-09-11 11:33:15.841167758 +0000 UTC m=+172.377074752" Sep 11 11:33:25.201156 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:25.201122 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:25.201543 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:25.201203 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:25.205693 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:25.205665 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:33:25.864922 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:33:25.864894 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:35:23.808421 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:23.808392 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 11:35:35.990888 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:35.990851 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb"] Sep 11 11:35:35.993513 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:35.993499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:35:35.996323 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:35.996300 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Sep 11 11:35:35.996435 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:35.996358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Sep 11 11:35:35.997546 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:35.997524 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-6dm97\"" Sep 11 11:35:36.003352 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:36.003332 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb"] Sep 11 11:35:36.075938 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:36.075906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwp6\" (UniqueName: \"kubernetes.io/projected/ce174939-e51d-448e-a9c3-e8e71eb15996-kube-api-access-nnwp6\") pod \"feast-operator-controller-manager-8c5f9c9d7-dhkvb\" (UID: \"ce174939-e51d-448e-a9c3-e8e71eb15996\") " pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:35:36.177106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:36.177081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwp6\" (UniqueName: \"kubernetes.io/projected/ce174939-e51d-448e-a9c3-e8e71eb15996-kube-api-access-nnwp6\") pod \"feast-operator-controller-manager-8c5f9c9d7-dhkvb\" (UID: \"ce174939-e51d-448e-a9c3-e8e71eb15996\") " pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:35:36.186459 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:36.186439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwp6\" (UniqueName: \"kubernetes.io/projected/ce174939-e51d-448e-a9c3-e8e71eb15996-kube-api-access-nnwp6\") pod \"feast-operator-controller-manager-8c5f9c9d7-dhkvb\" (UID: \"ce174939-e51d-448e-a9c3-e8e71eb15996\") " pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:35:36.304403 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:36.304324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:35:36.571572 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:36.571548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb"] Sep 11 11:35:36.574217 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:35:36.574192 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce174939_e51d_448e_a9c3_e8e71eb15996.slice/crio-ce94e22323fee704963fb6aa7e2bc75aaf00eda71fdcfdc66cd367ef302ea3cc WatchSource:0}: Error finding container ce94e22323fee704963fb6aa7e2bc75aaf00eda71fdcfdc66cd367ef302ea3cc: Status 404 returned error can't find the container with id ce94e22323fee704963fb6aa7e2bc75aaf00eda71fdcfdc66cd367ef302ea3cc Sep 11 11:35:36.576026 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:36.576013 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:35:37.333519 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:37.333482 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" event={"ID":"ce174939-e51d-448e-a9c3-e8e71eb15996","Type":"ContainerStarted","Data":"ce94e22323fee704963fb6aa7e2bc75aaf00eda71fdcfdc66cd367ef302ea3cc"} Sep 11 11:35:39.343789 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:39.343682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" event={"ID":"ce174939-e51d-448e-a9c3-e8e71eb15996","Type":"ContainerStarted","Data":"af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049"} Sep 11 11:35:39.344121 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:39.343797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:35:39.360478 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:39.360437 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" podStartSLOduration=2.067799037 podStartE2EDuration="4.360414616s" totalImagesPullingTime="2.292615579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:39.359074915 +0000 UTC m=+315.894981913" watchObservedRunningTime="2026-09-11 11:35:39.360414616 +0000 UTC m=+315.896321614" Sep 11 11:35:50.352273 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:35:50.352243 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:36:15.077099 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.077063 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb"] Sep 11 11:36:15.083872 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.083848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.086645 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.086626 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 11:36:15.086788 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.086679 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 11:36:15.086975 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.086959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-f7crl\"" Sep 11 11:36:15.087184 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.087149 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 11:36:15.087423 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.087407 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 11:36:15.087990 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.087970 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 11:36:15.087990 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.087978 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 11 11:36:15.088115 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.087978 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 11:36:15.094932 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.094911 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb"] Sep 11 11:36:15.257158 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.257132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-online-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.257158 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.257161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-offline-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.257364 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.257183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-ui-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.257364 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.257236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.257364 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.257276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0faf836-d475-4163-8990-815e96a91654-feast-data\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.257364 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.257334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7t5q\" (UniqueName: \"kubernetes.io/projected/e0faf836-d475-4163-8990-815e96a91654-kube-api-access-m7t5q\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.357916 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.357890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-online-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.358029 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.357920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-offline-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.358029 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.357942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-ui-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.358029 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.358014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.358164 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.358056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0faf836-d475-4163-8990-815e96a91654-feast-data\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.358164 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.358111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7t5q\" (UniqueName: \"kubernetes.io/projected/e0faf836-d475-4163-8990-815e96a91654-kube-api-access-m7t5q\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.358262 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:36:15.358189 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 11 11:36:15.358317 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:36:15.358271 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls podName:e0faf836-d475-4163-8990-815e96a91654 nodeName:}" failed. No retries permitted until 2026-09-11 11:36:15.858250847 +0000 UTC m=+352.394157837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls") pod "feast-simple-feast-setup-84664f6fc9-pscqb" (UID: "e0faf836-d475-4163-8990-815e96a91654") : secret "feast-simple-feast-setup-registry-tls" not found Sep 11 11:36:15.358479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.358457 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0faf836-d475-4163-8990-815e96a91654-feast-data\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.360612 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.360596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-offline-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.360700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.360627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-ui-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.360700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.360663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-online-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.369612 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.369583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7t5q\" (UniqueName: \"kubernetes.io/projected/e0faf836-d475-4163-8990-815e96a91654-kube-api-access-m7t5q\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.862662 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:15.862628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:15.862854 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:36:15.862808 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 11 11:36:15.862916 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:36:15.862888 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls podName:e0faf836-d475-4163-8990-815e96a91654 nodeName:}" failed. No retries permitted until 2026-09-11 11:36:16.86286969 +0000 UTC m=+353.398776666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls") pod "feast-simple-feast-setup-84664f6fc9-pscqb" (UID: "e0faf836-d475-4163-8990-815e96a91654") : secret "feast-simple-feast-setup-registry-tls" not found Sep 11 11:36:16.869597 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:16.869561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:16.872091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:16.872069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-pscqb\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:16.893185 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:16.893151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:17.167961 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:17.167929 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb"] Sep 11 11:36:17.169057 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:36:17.169018 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0faf836_d475_4163_8990_815e96a91654.slice/crio-62132b90bdfdc94a2d1e200ee132aa82cb487247618eb8ea8f21eefdada5071f WatchSource:0}: Error finding container 62132b90bdfdc94a2d1e200ee132aa82cb487247618eb8ea8f21eefdada5071f: Status 404 returned error can't find the container with id 62132b90bdfdc94a2d1e200ee132aa82cb487247618eb8ea8f21eefdada5071f Sep 11 11:36:17.491919 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:17.491836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerStarted","Data":"62132b90bdfdc94a2d1e200ee132aa82cb487247618eb8ea8f21eefdada5071f"} Sep 11 11:36:41.595208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:41.595163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerStarted","Data":"927a679db9ada3ee8a5455840a19d75e2b2aae1d310b6b7a19ee2ebaece5aeee"} Sep 11 11:36:46.615447 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:46.615409 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0faf836-d475-4163-8990-815e96a91654" containerID="927a679db9ada3ee8a5455840a19d75e2b2aae1d310b6b7a19ee2ebaece5aeee" exitCode=0 Sep 11 11:36:46.615852 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:46.615459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerDied","Data":"927a679db9ada3ee8a5455840a19d75e2b2aae1d310b6b7a19ee2ebaece5aeee"} Sep 11 11:36:47.621662 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:47.621624 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerStarted","Data":"608cb9d7d9a04757cc0b906dd43880a9298f67d8d9dd50c9d8f7260edc34ba33"} Sep 11 11:36:48.052362 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:48.052273 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59cd4c9dd6-mvwtb"] Sep 11 11:36:52.644316 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:52.644275 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0faf836-d475-4163-8990-815e96a91654" containerID="608cb9d7d9a04757cc0b906dd43880a9298f67d8d9dd50c9d8f7260edc34ba33" exitCode=0 Sep 11 11:36:52.644792 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:52.644342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerDied","Data":"608cb9d7d9a04757cc0b906dd43880a9298f67d8d9dd50c9d8f7260edc34ba33"} Sep 11 11:36:52.652427 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:52.652397 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 11 11:36:53.654227 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:53.654188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerStarted","Data":"bd8dd1e763f235edd2cafec3f081299105773d205e902fec8045fb1e31028f18"} Sep 11 11:36:53.654752 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:53.654254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerStarted","Data":"b745f7663fa48614c20e7d679acab955c571cb97f9173e94413521b67e3344ec"} Sep 11 11:36:53.654752 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:53.654274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerStarted","Data":"47426a2d75fb3d4671da458293258c4b495f4dce354646c4d39a738e34098962"} Sep 11 11:36:53.654752 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:53.654295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerStarted","Data":"89ec83ae92c8e0c85d6e64a3be7f1ca505742ba33cbfaeb64aa59d347f3c60cc"} Sep 11 11:36:53.685786 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:53.685696 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podStartSLOduration=14.419756463 podStartE2EDuration="38.685674518s" totalImagesPullingTime="24.265918055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:36:53.68548908 +0000 UTC m=+390.221396079" watchObservedRunningTime="2026-09-11 11:36:53.685674518 +0000 UTC m=+390.221581516" Sep 11 11:36:55.893605 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.893558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:55.894123 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.893631 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:55.894123 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.893653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:55.894123 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.893671 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:55.905403 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.896995 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 11 11:36:55.905403 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.897033 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 11 11:36:55.905403 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.897509 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Sep 11 11:36:55.905403 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:55.897788 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Sep 11 11:36:56.894159 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:56.894117 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:56.894738 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:56.894694 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:56.894889 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:56.894757 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:56.894889 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:56.894781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:36:58.893764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:58.893695 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.24:8816: connect: connection refused" Sep 11 11:36:58.893764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:58.893732 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 11 11:36:58.894325 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:58.893695 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.24:6571: connect: connection refused" Sep 11 11:36:58.894325 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:36:58.893921 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 11 11:37:01.899347 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:01.899320 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:01.899786 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:01.899444 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:01.899786 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:01.899672 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:01.900157 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:01.900137 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:01.900546 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:01.900526 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:01.900623 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:01.900593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:01.900687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:01.900672 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:02.703122 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:02.703092 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:37:13.075731 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.075670 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59cd4c9dd6-mvwtb" podUID="e302366f-e71d-4a9b-aa89-d353529a0dcc" containerName="console" containerID="cri-o://2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263" gracePeriod=15 Sep 11 11:37:13.350143 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.350119 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:37:13.444765 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.444740 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqgq8\" (UniqueName: \"kubernetes.io/projected/e302366f-e71d-4a9b-aa89-d353529a0dcc-kube-api-access-qqgq8\") pod \"e302366f-e71d-4a9b-aa89-d353529a0dcc\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " Sep 11 11:37:13.444901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.444781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-service-ca\") pod \"e302366f-e71d-4a9b-aa89-d353529a0dcc\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " Sep 11 11:37:13.444901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.444828 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-oauth-config\") pod \"e302366f-e71d-4a9b-aa89-d353529a0dcc\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " Sep 11 11:37:13.444901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.444853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-serving-cert\") pod \"e302366f-e71d-4a9b-aa89-d353529a0dcc\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " Sep 11 11:37:13.444901 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.444878 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-config\") pod \"e302366f-e71d-4a9b-aa89-d353529a0dcc\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " Sep 11 11:37:13.445102 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.444907 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-trusted-ca-bundle\") pod \"e302366f-e71d-4a9b-aa89-d353529a0dcc\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " Sep 11 11:37:13.445102 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.444937 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-oauth-serving-cert\") pod \"e302366f-e71d-4a9b-aa89-d353529a0dcc\" (UID: \"e302366f-e71d-4a9b-aa89-d353529a0dcc\") " Sep 11 11:37:13.445371 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.445250 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-service-ca" (OuterVolumeSpecName: "service-ca") pod "e302366f-e71d-4a9b-aa89-d353529a0dcc" (UID: "e302366f-e71d-4a9b-aa89-d353529a0dcc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:37:13.445482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.445409 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e302366f-e71d-4a9b-aa89-d353529a0dcc" (UID: "e302366f-e71d-4a9b-aa89-d353529a0dcc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:37:13.445482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.445421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e302366f-e71d-4a9b-aa89-d353529a0dcc" (UID: "e302366f-e71d-4a9b-aa89-d353529a0dcc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:37:13.445482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.445415 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-config" (OuterVolumeSpecName: "console-config") pod "e302366f-e71d-4a9b-aa89-d353529a0dcc" (UID: "e302366f-e71d-4a9b-aa89-d353529a0dcc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:37:13.447059 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.447024 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e302366f-e71d-4a9b-aa89-d353529a0dcc-kube-api-access-qqgq8" (OuterVolumeSpecName: "kube-api-access-qqgq8") pod "e302366f-e71d-4a9b-aa89-d353529a0dcc" (UID: "e302366f-e71d-4a9b-aa89-d353529a0dcc"). InnerVolumeSpecName "kube-api-access-qqgq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:37:13.447514 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.447487 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e302366f-e71d-4a9b-aa89-d353529a0dcc" (UID: "e302366f-e71d-4a9b-aa89-d353529a0dcc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:37:13.447627 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.447550 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e302366f-e71d-4a9b-aa89-d353529a0dcc" (UID: "e302366f-e71d-4a9b-aa89-d353529a0dcc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:37:13.546058 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.546038 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-oauth-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:37:13.546058 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.546057 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:37:13.546174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.546068 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-console-config\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:37:13.546174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.546076 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-trusted-ca-bundle\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:37:13.546174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.546086 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-oauth-serving-cert\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:37:13.546174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.546094 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqgq8\" (UniqueName: \"kubernetes.io/projected/e302366f-e71d-4a9b-aa89-d353529a0dcc-kube-api-access-qqgq8\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:37:13.546174 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.546103 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e302366f-e71d-4a9b-aa89-d353529a0dcc-service-ca\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:37:13.744848 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.744773 2579 generic.go:356] "Generic (PLEG): container finished" podID="e302366f-e71d-4a9b-aa89-d353529a0dcc" containerID="2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263" exitCode=0 Sep 11 11:37:13.744848 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.744820 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59cd4c9dd6-mvwtb" Sep 11 11:37:13.744848 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.744836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cd4c9dd6-mvwtb" event={"ID":"e302366f-e71d-4a9b-aa89-d353529a0dcc","Type":"ContainerDied","Data":"2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263"} Sep 11 11:37:13.745054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.744882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59cd4c9dd6-mvwtb" event={"ID":"e302366f-e71d-4a9b-aa89-d353529a0dcc","Type":"ContainerDied","Data":"947ffe157303ab4b0834ea0575bd03592e9722308bf746b2230a51f571a5491e"} Sep 11 11:37:13.745054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.744909 2579 scope.go:117] "RemoveContainer" containerID="2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263" Sep 11 11:37:13.753213 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.753196 2579 scope.go:117] "RemoveContainer" containerID="2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263" Sep 11 11:37:13.753508 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:37:13.753489 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263\": container with ID starting with 2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263 not found: ID does not exist" containerID="2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263" Sep 11 11:37:13.753550 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.753518 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263"} err="failed to get container status \"2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263\": rpc error: code = NotFound desc = could not find container \"2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263\": container with ID starting with 2a5b86fdde21a078d8f0257c77541b768020c6d5649d8d56d9bfb298597ee263 not found: ID does not exist" Sep 11 11:37:13.768807 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.768784 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59cd4c9dd6-mvwtb"] Sep 11 11:37:13.770249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.770228 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59cd4c9dd6-mvwtb"] Sep 11 11:37:13.904627 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:37:13.904599 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e302366f-e71d-4a9b-aa89-d353529a0dcc" path="/var/lib/kubelet/pods/e302366f-e71d-4a9b-aa89-d353529a0dcc/volumes" Sep 11 11:38:05.561252 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.561215 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb"] Sep 11 11:38:05.561685 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.561465 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="registry" containerID="cri-o://89ec83ae92c8e0c85d6e64a3be7f1ca505742ba33cbfaeb64aa59d347f3c60cc" gracePeriod=30 Sep 11 11:38:05.561685 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.561507 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" containerID="cri-o://bd8dd1e763f235edd2cafec3f081299105773d205e902fec8045fb1e31028f18" gracePeriod=30 Sep 11 11:38:05.561685 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.561531 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" containerID="cri-o://47426a2d75fb3d4671da458293258c4b495f4dce354646c4d39a738e34098962" gracePeriod=30 Sep 11 11:38:05.561685 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.561534 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="offline" containerID="cri-o://b745f7663fa48614c20e7d679acab955c571cb97f9173e94413521b67e3344ec" gracePeriod=30 Sep 11 11:38:05.964768 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.964706 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr"] Sep 11 11:38:05.965189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.965166 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e302366f-e71d-4a9b-aa89-d353529a0dcc" containerName="console" Sep 11 11:38:05.965272 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.965192 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e302366f-e71d-4a9b-aa89-d353529a0dcc" containerName="console" Sep 11 11:38:05.965272 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.965264 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e302366f-e71d-4a9b-aa89-d353529a0dcc" containerName="console" Sep 11 11:38:05.969346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.969324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:05.972505 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.972472 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-qklwh\"" Sep 11 11:38:05.991602 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:05.991571 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr"] Sep 11 11:38:06.027501 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.027466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-offline-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.027501 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.027511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzv7\" (UniqueName: \"kubernetes.io/projected/177baa81-038e-4a6d-89c4-cdad61b27e51-kube-api-access-xbzv7\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.027731 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.027545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-online-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.027731 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.027618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.027731 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.027640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-ui-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.027731 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.027676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/177baa81-038e-4a6d-89c4-cdad61b27e51-feast-data\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.128765 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.128708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-offline-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.128765 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.128763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzv7\" (UniqueName: \"kubernetes.io/projected/177baa81-038e-4a6d-89c4-cdad61b27e51-kube-api-access-xbzv7\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.128976 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.128789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-online-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.128976 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.128927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.128976 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.128959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-ui-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.129128 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.129022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/177baa81-038e-4a6d-89c4-cdad61b27e51-feast-data\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.129414 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.129393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/177baa81-038e-4a6d-89c4-cdad61b27e51-feast-data\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.131561 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.131530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-registry-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.131856 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.131834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-offline-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.132023 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.132003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-online-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.132090 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.132063 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-ui-tls\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.138108 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.138087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzv7\" (UniqueName: \"kubernetes.io/projected/177baa81-038e-4a6d-89c4-cdad61b27e51-kube-api-access-xbzv7\") pod \"feast-simple-feast-setup-84664f6fc9-hhlvr\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.279155 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.279058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:06.560283 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.560254 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr"] Sep 11 11:38:06.562401 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:38:06.562377 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod177baa81_038e_4a6d_89c4_cdad61b27e51.slice/crio-af4cc12c9cf6569eeed29ed6569f58943c9d81f53dcb9efdfcc8f93c947525d9 WatchSource:0}: Error finding container af4cc12c9cf6569eeed29ed6569f58943c9d81f53dcb9efdfcc8f93c947525d9: Status 404 returned error can't find the container with id af4cc12c9cf6569eeed29ed6569f58943c9d81f53dcb9efdfcc8f93c947525d9 Sep 11 11:38:06.954530 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.954497 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0faf836-d475-4163-8990-815e96a91654" containerID="bd8dd1e763f235edd2cafec3f081299105773d205e902fec8045fb1e31028f18" exitCode=0 Sep 11 11:38:06.954530 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.954520 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0faf836-d475-4163-8990-815e96a91654" containerID="47426a2d75fb3d4671da458293258c4b495f4dce354646c4d39a738e34098962" exitCode=0 Sep 11 11:38:06.954799 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.954569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerDied","Data":"bd8dd1e763f235edd2cafec3f081299105773d205e902fec8045fb1e31028f18"} Sep 11 11:38:06.954799 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.954610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerDied","Data":"47426a2d75fb3d4671da458293258c4b495f4dce354646c4d39a738e34098962"} Sep 11 11:38:06.956090 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.956068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerStarted","Data":"3f7aa227dfc982df91a94c2b46c3b5dac0e82f530ba44b8d5950ff92ff04ebf2"} Sep 11 11:38:06.956223 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:06.956100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerStarted","Data":"af4cc12c9cf6569eeed29ed6569f58943c9d81f53dcb9efdfcc8f93c947525d9"} Sep 11 11:38:11.900889 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:11.900845 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 11 11:38:11.976499 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:11.976464 2579 generic.go:356] "Generic (PLEG): container finished" podID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerID="3f7aa227dfc982df91a94c2b46c3b5dac0e82f530ba44b8d5950ff92ff04ebf2" exitCode=0 Sep 11 11:38:11.976651 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:11.976534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerDied","Data":"3f7aa227dfc982df91a94c2b46c3b5dac0e82f530ba44b8d5950ff92ff04ebf2"} Sep 11 11:38:12.700621 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:12.700575 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 11 11:38:12.982206 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:12.982114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerStarted","Data":"27ef693f421139c5e30f06f0c8e9d69924218100c21f46e6adf21b2a3cd8db33"} Sep 11 11:38:18.003766 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:18.003662 2579 generic.go:356] "Generic (PLEG): container finished" podID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerID="27ef693f421139c5e30f06f0c8e9d69924218100c21f46e6adf21b2a3cd8db33" exitCode=0 Sep 11 11:38:18.003766 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:18.003738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerDied","Data":"27ef693f421139c5e30f06f0c8e9d69924218100c21f46e6adf21b2a3cd8db33"} Sep 11 11:38:19.014465 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:19.014418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerStarted","Data":"67b39e47c84096d4074b8da1cae5c2b8a3c4d673cef606bfe857f31d4a207964"} Sep 11 11:38:19.014979 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:19.014490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerStarted","Data":"d5189f10ffa923de2211094c999f2869af8c0f8de1bd0994fb077512f5a0ef5e"} Sep 11 11:38:19.014979 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:19.014514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerStarted","Data":"189f6a124d4237e541c0bfff4fb1c96a7441fdc17dbcd6e17981743f3476298c"} Sep 11 11:38:19.014979 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:19.014534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerStarted","Data":"d88ed211e7017b186d40b4f8880e773ebbf7d7b7263cd9464f526391e3a21add"} Sep 11 11:38:19.077795 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:19.077705 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podStartSLOduration=14.077683166 podStartE2EDuration="14.077683166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:38:19.073849018 +0000 UTC m=+475.609756018" watchObservedRunningTime="2026-09-11 11:38:19.077683166 +0000 UTC m=+475.613590163" Sep 11 11:38:21.279883 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.279846 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:21.280366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.279913 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:21.280366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.279935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:21.280366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.279955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:21.281937 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.281898 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Sep 11 11:38:21.281937 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.281923 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused" Sep 11 11:38:21.282152 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.281928 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 11 11:38:21.282152 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.281984 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 11 11:38:21.901108 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:21.901063 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 11 11:38:22.701159 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:22.701108 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 11 11:38:24.280151 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:24.280098 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.25:8816: connect: connection refused" Sep 11 11:38:24.280661 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:24.280098 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 11 11:38:24.280661 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:24.280124 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 11 11:38:24.280661 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:24.280588 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.25:6571: connect: connection refused" Sep 11 11:38:26.279340 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:26.279306 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:26.279692 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:26.279358 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:26.279692 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:26.279373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:26.279692 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:26.279384 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:27.281062 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:27.281030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:27.281521 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:27.281339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:27.281808 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:27.281784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:27.281945 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:27.281857 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:27.281945 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:27.281900 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:27.286456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:27.286434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:28.060762 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:28.060735 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:28.063652 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:28.063629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:38:31.901336 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:31.901297 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.24:8443: connect: connection refused" Sep 11 11:38:31.903547 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:31.903523 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:38:32.700255 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:32.700211 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" probeResult="failure" output="Get \"https://10.133.0.24:6567/health\": dial tcp 10.133.0.24:6567: connect: connection refused" Sep 11 11:38:32.700452 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:32.700330 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:38:36.094217 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.094186 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0faf836-d475-4163-8990-815e96a91654" containerID="b745f7663fa48614c20e7d679acab955c571cb97f9173e94413521b67e3344ec" exitCode=137 Sep 11 11:38:36.094217 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.094217 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0faf836-d475-4163-8990-815e96a91654" containerID="89ec83ae92c8e0c85d6e64a3be7f1ca505742ba33cbfaeb64aa59d347f3c60cc" exitCode=137 Sep 11 11:38:36.094777 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.094263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerDied","Data":"b745f7663fa48614c20e7d679acab955c571cb97f9173e94413521b67e3344ec"} Sep 11 11:38:36.094777 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.094308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerDied","Data":"89ec83ae92c8e0c85d6e64a3be7f1ca505742ba33cbfaeb64aa59d347f3c60cc"} Sep 11 11:38:36.203831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.203809 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:38:36.380586 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.380546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0faf836-d475-4163-8990-815e96a91654-feast-data\") pod \"e0faf836-d475-4163-8990-815e96a91654\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " Sep 11 11:38:36.380781 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.380611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls\") pod \"e0faf836-d475-4163-8990-815e96a91654\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " Sep 11 11:38:36.380781 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.380653 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-online-tls\") pod \"e0faf836-d475-4163-8990-815e96a91654\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " Sep 11 11:38:36.380781 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.380704 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-offline-tls\") pod \"e0faf836-d475-4163-8990-815e96a91654\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " Sep 11 11:38:36.380781 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.380755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-ui-tls\") pod \"e0faf836-d475-4163-8990-815e96a91654\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " Sep 11 11:38:36.380988 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.380787 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7t5q\" (UniqueName: \"kubernetes.io/projected/e0faf836-d475-4163-8990-815e96a91654-kube-api-access-m7t5q\") pod \"e0faf836-d475-4163-8990-815e96a91654\" (UID: \"e0faf836-d475-4163-8990-815e96a91654\") " Sep 11 11:38:36.381178 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.381146 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0faf836-d475-4163-8990-815e96a91654-feast-data" (OuterVolumeSpecName: "feast-data") pod "e0faf836-d475-4163-8990-815e96a91654" (UID: "e0faf836-d475-4163-8990-815e96a91654"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:38:36.383023 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.382985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e0faf836-d475-4163-8990-815e96a91654" (UID: "e0faf836-d475-4163-8990-815e96a91654"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:38:36.383023 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.383011 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0faf836-d475-4163-8990-815e96a91654-kube-api-access-m7t5q" (OuterVolumeSpecName: "kube-api-access-m7t5q") pod "e0faf836-d475-4163-8990-815e96a91654" (UID: "e0faf836-d475-4163-8990-815e96a91654"). InnerVolumeSpecName "kube-api-access-m7t5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:38:36.383140 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.383053 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "e0faf836-d475-4163-8990-815e96a91654" (UID: "e0faf836-d475-4163-8990-815e96a91654"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:38:36.383442 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.383419 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e0faf836-d475-4163-8990-815e96a91654" (UID: "e0faf836-d475-4163-8990-815e96a91654"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:38:36.383521 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.383446 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-online-tls" (OuterVolumeSpecName: "online-tls") pod "e0faf836-d475-4163-8990-815e96a91654" (UID: "e0faf836-d475-4163-8990-815e96a91654"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:38:36.482020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.481994 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e0faf836-d475-4163-8990-815e96a91654-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:38:36.482020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.482018 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:38:36.482158 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.482028 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:38:36.482158 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.482035 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:38:36.482158 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.482043 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/e0faf836-d475-4163-8990-815e96a91654-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:38:36.482158 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:36.482051 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7t5q\" (UniqueName: \"kubernetes.io/projected/e0faf836-d475-4163-8990-815e96a91654-kube-api-access-m7t5q\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:38:37.101431 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.101399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" event={"ID":"e0faf836-d475-4163-8990-815e96a91654","Type":"ContainerDied","Data":"62132b90bdfdc94a2d1e200ee132aa82cb487247618eb8ea8f21eefdada5071f"} Sep 11 11:38:37.101935 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.101449 2579 scope.go:117] "RemoveContainer" containerID="bd8dd1e763f235edd2cafec3f081299105773d205e902fec8045fb1e31028f18" Sep 11 11:38:37.101935 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.101412 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb" Sep 11 11:38:37.110076 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.110059 2579 scope.go:117] "RemoveContainer" containerID="b745f7663fa48614c20e7d679acab955c571cb97f9173e94413521b67e3344ec" Sep 11 11:38:37.117467 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.117447 2579 scope.go:117] "RemoveContainer" containerID="47426a2d75fb3d4671da458293258c4b495f4dce354646c4d39a738e34098962" Sep 11 11:38:37.125114 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.125099 2579 scope.go:117] "RemoveContainer" containerID="89ec83ae92c8e0c85d6e64a3be7f1ca505742ba33cbfaeb64aa59d347f3c60cc" Sep 11 11:38:37.125372 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.125354 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb"] Sep 11 11:38:37.129334 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.129314 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-pscqb"] Sep 11 11:38:37.133013 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.132999 2579 scope.go:117] "RemoveContainer" containerID="608cb9d7d9a04757cc0b906dd43880a9298f67d8d9dd50c9d8f7260edc34ba33" Sep 11 11:38:37.139897 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.139882 2579 scope.go:117] "RemoveContainer" containerID="927a679db9ada3ee8a5455840a19d75e2b2aae1d310b6b7a19ee2ebaece5aeee" Sep 11 11:38:37.898744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:38:37.898687 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0faf836-d475-4163-8990-815e96a91654" path="/var/lib/kubelet/pods/e0faf836-d475-4163-8990-815e96a91654/volumes" Sep 11 11:39:30.947323 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947289 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8"] Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947591 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="feast-init" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947604 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="feast-init" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947613 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="registry" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947619 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="registry" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947629 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947635 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947647 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947653 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947667 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="offline" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947671 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="offline" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947678 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="feast-apply" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947683 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="feast-apply" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947751 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="offline" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947761 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="ui" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947770 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="registry" Sep 11 11:39:30.947815 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.947779 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0faf836-d475-4163-8990-815e96a91654" containerName="online" Sep 11 11:39:30.951867 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.951850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:30.954757 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.954710 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 11:39:30.954866 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.954765 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-k2djh\"" Sep 11 11:39:30.955181 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.955168 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 11:39:30.956002 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.955981 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 11:39:30.956099 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.956005 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 11 11:39:30.956099 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.956005 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 11:39:30.956099 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.955982 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 11:39:30.956305 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.956269 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 11:39:30.967539 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:30.967521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8"] Sep 11 11:39:31.063164 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.063133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-registry-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.063270 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.063182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-feast-data\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.063270 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.063216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lh2\" (UniqueName: \"kubernetes.io/projected/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-kube-api-access-g4lh2\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.063270 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.063255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-offline-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.063373 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.063283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.063373 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.063317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-ui-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164078 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-registry-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-feast-data\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lh2\" (UniqueName: \"kubernetes.io/projected/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-kube-api-access-g4lh2\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-offline-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-ui-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164423 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:39:31.164263 2579 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 11:39:31.164423 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:39:31.164336 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls podName:5ea27f46-75a5-4a2f-a2fe-2f846db44b81 nodeName:}" failed. No retries permitted until 2026-09-11 11:39:31.66431504 +0000 UTC m=+548.200222031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls") pod "feast-simple-feast-remote-setup-7cd844c557-ntll8" (UID: "5ea27f46-75a5-4a2f-a2fe-2f846db44b81") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 11 11:39:31.164538 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-feast-data\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.164867 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.164847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-registry-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.166651 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.166626 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-offline-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.166803 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.166784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-ui-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.175244 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.175221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lh2\" (UniqueName: \"kubernetes.io/projected/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-kube-api-access-g4lh2\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.668451 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.668411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.670912 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.670886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls\") pod \"feast-simple-feast-remote-setup-7cd844c557-ntll8\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:31.860880 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:31.860850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:39:32.136159 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:32.136136 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8"] Sep 11 11:39:32.137331 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:39:32.137302 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ea27f46_75a5_4a2f_a2fe_2f846db44b81.slice/crio-bc490a3a41e250f3fe63f8a6aafba751ce280e1883c0b01d081aa09f9fef4ff4 WatchSource:0}: Error finding container bc490a3a41e250f3fe63f8a6aafba751ce280e1883c0b01d081aa09f9fef4ff4: Status 404 returned error can't find the container with id bc490a3a41e250f3fe63f8a6aafba751ce280e1883c0b01d081aa09f9fef4ff4 Sep 11 11:39:32.319457 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:32.319420 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerStarted","Data":"3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec"} Sep 11 11:39:32.319637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:32.319467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerStarted","Data":"bc490a3a41e250f3fe63f8a6aafba751ce280e1883c0b01d081aa09f9fef4ff4"} Sep 11 11:39:37.339613 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:37.339531 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerID="3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec" exitCode=0 Sep 11 11:39:37.339613 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:37.339574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerDied","Data":"3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec"} Sep 11 11:39:38.350560 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:39:38.350522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerStarted","Data":"660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee"} Sep 11 11:40:01.448560 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:01.448525 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerID="660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee" exitCode=0 Sep 11 11:40:01.448988 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:01.448590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerDied","Data":"660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee"} Sep 11 11:40:02.457379 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:02.457336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerStarted","Data":"e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa"} Sep 11 11:40:02.457806 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:02.457403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerStarted","Data":"acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a"} Sep 11 11:40:02.457806 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:02.457426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerStarted","Data":"6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296"} Sep 11 11:40:02.491346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:02.491292 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podStartSLOduration=32.491277606 podStartE2EDuration="32.491277606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:40:02.490228591 +0000 UTC m=+579.026135592" watchObservedRunningTime="2026-09-11 11:40:02.491277606 +0000 UTC m=+579.027184602" Sep 11 11:40:04.861636 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:04.861591 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:04.862055 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:04.861653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:04.862055 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:04.861674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:04.863188 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:04.863153 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 11 11:40:04.863289 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:04.863167 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused" Sep 11 11:40:04.863289 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:04.863204 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 11 11:40:07.862016 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:07.861964 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 11 11:40:07.862016 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:07.861988 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 11 11:40:07.862510 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:07.861965 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.26:8816: connect: connection refused" Sep 11 11:40:10.861959 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:10.861925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:10.862422 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:10.862098 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:10.862634 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:10.862599 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:10.862744 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:10.862642 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:10.862810 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:10.862795 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:10.863109 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:10.863096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:10.883209 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:10.883188 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:11.498053 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:11.498025 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:40:11.517091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:40:11.517066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:41:13.288412 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.288380 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8"] Sep 11 11:41:13.288936 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.288621 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" containerID="cri-o://6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296" gracePeriod=30 Sep 11 11:41:13.288936 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.288692 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="offline" containerID="cri-o://acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a" gracePeriod=30 Sep 11 11:41:13.288936 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.288680 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" containerID="cri-o://e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa" gracePeriod=30 Sep 11 11:41:13.401960 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.401929 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr"] Sep 11 11:41:13.402374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.402174 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="registry" containerID="cri-o://d88ed211e7017b186d40b4f8880e773ebbf7d7b7263cd9464f526391e3a21add" gracePeriod=30 Sep 11 11:41:13.402374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.402227 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="offline" containerID="cri-o://d5189f10ffa923de2211094c999f2869af8c0f8de1bd0994fb077512f5a0ef5e" gracePeriod=30 Sep 11 11:41:13.402374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.402247 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" containerID="cri-o://189f6a124d4237e541c0bfff4fb1c96a7441fdc17dbcd6e17981743f3476298c" gracePeriod=30 Sep 11 11:41:13.402374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:13.402259 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" containerID="cri-o://67b39e47c84096d4074b8da1cae5c2b8a3c4d673cef606bfe857f31d4a207964" gracePeriod=30 Sep 11 11:41:14.744456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.744425 2579 generic.go:356] "Generic (PLEG): container finished" podID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerID="67b39e47c84096d4074b8da1cae5c2b8a3c4d673cef606bfe857f31d4a207964" exitCode=0 Sep 11 11:41:14.744456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.744453 2579 generic.go:356] "Generic (PLEG): container finished" podID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerID="189f6a124d4237e541c0bfff4fb1c96a7441fdc17dbcd6e17981743f3476298c" exitCode=0 Sep 11 11:41:14.744911 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.744500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerDied","Data":"67b39e47c84096d4074b8da1cae5c2b8a3c4d673cef606bfe857f31d4a207964"} Sep 11 11:41:14.744911 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.744545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerDied","Data":"189f6a124d4237e541c0bfff4fb1c96a7441fdc17dbcd6e17981743f3476298c"} Sep 11 11:41:14.747101 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.747078 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerID="e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa" exitCode=0 Sep 11 11:41:14.747101 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.747098 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerID="6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296" exitCode=0 Sep 11 11:41:14.747257 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.747115 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerDied","Data":"e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa"} Sep 11 11:41:14.747257 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:14.747139 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerDied","Data":"6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296"} Sep 11 11:41:17.282037 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:17.281994 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 11 11:41:18.060469 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:18.060428 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 11 11:41:20.863130 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:20.863086 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 11 11:41:21.498218 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:21.498180 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 11 11:41:27.282427 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:27.282376 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 11 11:41:28.060788 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:28.060753 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 11 11:41:30.863168 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:30.863125 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 11 11:41:31.498849 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:31.498809 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 11 11:41:37.281817 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:37.281778 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.25:8443: connect: connection refused" Sep 11 11:41:37.282224 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:37.281911 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:41:38.060947 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:38.060897 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" probeResult="failure" output="Get \"https://10.133.0.25:6567/health\": dial tcp 10.133.0.25:6567: connect: connection refused" Sep 11 11:41:38.061136 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:38.061043 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:41:40.862812 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:40.862768 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.26:8443: connect: connection refused" Sep 11 11:41:40.863234 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:40.862892 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:41:41.498727 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:41.498689 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" probeResult="failure" output="Get \"https://10.133.0.26:6567/health\": dial tcp 10.133.0.26:6567: connect: connection refused" Sep 11 11:41:41.498874 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:41.498821 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:41:43.503527 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.503499 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:41:43.548652 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.548567 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-feast-data\") pod \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " Sep 11 11:41:43.548652 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.548633 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-offline-tls\") pod \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " Sep 11 11:41:43.548907 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.548665 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-ui-tls\") pod \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " Sep 11 11:41:43.548907 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.548685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4lh2\" (UniqueName: \"kubernetes.io/projected/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-kube-api-access-g4lh2\") pod \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " Sep 11 11:41:43.548907 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.548705 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls\") pod \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " Sep 11 11:41:43.548907 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.548800 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-registry-tls\") pod \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\" (UID: \"5ea27f46-75a5-4a2f-a2fe-2f846db44b81\") " Sep 11 11:41:43.549237 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.549206 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5ea27f46-75a5-4a2f-a2fe-2f846db44b81" (UID: "5ea27f46-75a5-4a2f-a2fe-2f846db44b81"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:41:43.549352 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.549255 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-feast-data" (OuterVolumeSpecName: "feast-data") pod "5ea27f46-75a5-4a2f-a2fe-2f846db44b81" (UID: "5ea27f46-75a5-4a2f-a2fe-2f846db44b81"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:43.551630 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.551601 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5ea27f46-75a5-4a2f-a2fe-2f846db44b81" (UID: "5ea27f46-75a5-4a2f-a2fe-2f846db44b81"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:41:43.551630 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.551615 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5ea27f46-75a5-4a2f-a2fe-2f846db44b81" (UID: "5ea27f46-75a5-4a2f-a2fe-2f846db44b81"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:41:43.551791 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.551681 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls" (OuterVolumeSpecName: "online-tls") pod "5ea27f46-75a5-4a2f-a2fe-2f846db44b81" (UID: "5ea27f46-75a5-4a2f-a2fe-2f846db44b81"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:41:43.551791 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.551679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-kube-api-access-g4lh2" (OuterVolumeSpecName: "kube-api-access-g4lh2") pod "5ea27f46-75a5-4a2f-a2fe-2f846db44b81" (UID: "5ea27f46-75a5-4a2f-a2fe-2f846db44b81"). InnerVolumeSpecName "kube-api-access-g4lh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:41:43.650253 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.650220 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:43.650253 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.650245 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:43.650253 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.650254 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:43.650479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.650262 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4lh2\" (UniqueName: \"kubernetes.io/projected/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-kube-api-access-g4lh2\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:43.650479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.650285 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:43.650479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.650294 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/5ea27f46-75a5-4a2f-a2fe-2f846db44b81-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:43.858232 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.858200 2579 generic.go:356] "Generic (PLEG): container finished" podID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerID="d5189f10ffa923de2211094c999f2869af8c0f8de1bd0994fb077512f5a0ef5e" exitCode=137 Sep 11 11:41:43.858232 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.858222 2579 generic.go:356] "Generic (PLEG): container finished" podID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerID="d88ed211e7017b186d40b4f8880e773ebbf7d7b7263cd9464f526391e3a21add" exitCode=137 Sep 11 11:41:43.858413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.858267 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerDied","Data":"d5189f10ffa923de2211094c999f2869af8c0f8de1bd0994fb077512f5a0ef5e"} Sep 11 11:41:43.858413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.858311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerDied","Data":"d88ed211e7017b186d40b4f8880e773ebbf7d7b7263cd9464f526391e3a21add"} Sep 11 11:41:43.860868 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.860848 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerID="acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a" exitCode=137 Sep 11 11:41:43.860969 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.860875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerDied","Data":"acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a"} Sep 11 11:41:43.860969 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.860899 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" event={"ID":"5ea27f46-75a5-4a2f-a2fe-2f846db44b81","Type":"ContainerDied","Data":"bc490a3a41e250f3fe63f8a6aafba751ce280e1883c0b01d081aa09f9fef4ff4"} Sep 11 11:41:43.860969 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.860908 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8" Sep 11 11:41:43.860969 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.860918 2579 scope.go:117] "RemoveContainer" containerID="e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa" Sep 11 11:41:43.869390 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.869373 2579 scope.go:117] "RemoveContainer" containerID="acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a" Sep 11 11:41:43.876938 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.876918 2579 scope.go:117] "RemoveContainer" containerID="6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296" Sep 11 11:41:43.883391 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.883368 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8"] Sep 11 11:41:43.884544 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.884529 2579 scope.go:117] "RemoveContainer" containerID="660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee" Sep 11 11:41:43.887416 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.887393 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7cd844c557-ntll8"] Sep 11 11:41:43.892578 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.892563 2579 scope.go:117] "RemoveContainer" containerID="3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec" Sep 11 11:41:43.899648 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.899623 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" path="/var/lib/kubelet/pods/5ea27f46-75a5-4a2f-a2fe-2f846db44b81/volumes" Sep 11 11:41:43.900038 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.900027 2579 scope.go:117] "RemoveContainer" containerID="e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa" Sep 11 11:41:43.900298 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:41:43.900282 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa\": container with ID starting with e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa not found: ID does not exist" containerID="e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa" Sep 11 11:41:43.900349 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.900306 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa"} err="failed to get container status \"e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa\": rpc error: code = NotFound desc = could not find container \"e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa\": container with ID starting with e36b1d4eaa4f5948313fc5e9b9a3d739becf915308731b6ca14d7e9c8d7a25fa not found: ID does not exist" Sep 11 11:41:43.900349 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.900326 2579 scope.go:117] "RemoveContainer" containerID="acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a" Sep 11 11:41:43.900571 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:41:43.900553 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a\": container with ID starting with acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a not found: ID does not exist" containerID="acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a" Sep 11 11:41:43.900635 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.900579 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a"} err="failed to get container status \"acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a\": rpc error: code = NotFound desc = could not find container \"acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a\": container with ID starting with acc26c90bdc78bdb1c1415411e79b4d6234ec9c43c536b17f931897b5bf3291a not found: ID does not exist" Sep 11 11:41:43.900635 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.900601 2579 scope.go:117] "RemoveContainer" containerID="6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296" Sep 11 11:41:43.900845 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:41:43.900830 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296\": container with ID starting with 6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296 not found: ID does not exist" containerID="6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296" Sep 11 11:41:43.900894 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.900849 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296"} err="failed to get container status \"6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296\": rpc error: code = NotFound desc = could not find container \"6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296\": container with ID starting with 6ef420ee7d5a8fb60e046e666d1a783ad99c7ac4f71693671b88b4eb30f5a296 not found: ID does not exist" Sep 11 11:41:43.900894 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.900861 2579 scope.go:117] "RemoveContainer" containerID="660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee" Sep 11 11:41:43.901103 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:41:43.901084 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee\": container with ID starting with 660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee not found: ID does not exist" containerID="660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee" Sep 11 11:41:43.901165 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.901109 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee"} err="failed to get container status \"660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee\": rpc error: code = NotFound desc = could not find container \"660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee\": container with ID starting with 660cba8672744052a255d90797db99c01c9286686ffb91b45e1a428b303f6fee not found: ID does not exist" Sep 11 11:41:43.901165 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.901128 2579 scope.go:117] "RemoveContainer" containerID="3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec" Sep 11 11:41:43.901355 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:41:43.901340 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec\": container with ID starting with 3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec not found: ID does not exist" containerID="3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec" Sep 11 11:41:43.901393 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:43.901358 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec"} err="failed to get container status \"3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec\": rpc error: code = NotFound desc = could not find container \"3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec\": container with ID starting with 3124b91f575bb299ed663bcd1ec93e19927359a6b4f70b5e648b3b72a33d43ec not found: ID does not exist" Sep 11 11:41:44.044152 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.044130 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:41:44.155243 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.155172 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-online-tls\") pod \"177baa81-038e-4a6d-89c4-cdad61b27e51\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " Sep 11 11:41:44.155243 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.155223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-offline-tls\") pod \"177baa81-038e-4a6d-89c4-cdad61b27e51\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " Sep 11 11:41:44.155243 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.155241 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/177baa81-038e-4a6d-89c4-cdad61b27e51-feast-data\") pod \"177baa81-038e-4a6d-89c4-cdad61b27e51\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " Sep 11 11:41:44.155504 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.155268 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbzv7\" (UniqueName: \"kubernetes.io/projected/177baa81-038e-4a6d-89c4-cdad61b27e51-kube-api-access-xbzv7\") pod \"177baa81-038e-4a6d-89c4-cdad61b27e51\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " Sep 11 11:41:44.155504 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.155287 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-registry-tls\") pod \"177baa81-038e-4a6d-89c4-cdad61b27e51\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " Sep 11 11:41:44.155504 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.155311 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-ui-tls\") pod \"177baa81-038e-4a6d-89c4-cdad61b27e51\" (UID: \"177baa81-038e-4a6d-89c4-cdad61b27e51\") " Sep 11 11:41:44.155804 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.155777 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177baa81-038e-4a6d-89c4-cdad61b27e51-feast-data" (OuterVolumeSpecName: "feast-data") pod "177baa81-038e-4a6d-89c4-cdad61b27e51" (UID: "177baa81-038e-4a6d-89c4-cdad61b27e51"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:44.157386 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.157365 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-online-tls" (OuterVolumeSpecName: "online-tls") pod "177baa81-038e-4a6d-89c4-cdad61b27e51" (UID: "177baa81-038e-4a6d-89c4-cdad61b27e51"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:41:44.157933 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.157902 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177baa81-038e-4a6d-89c4-cdad61b27e51-kube-api-access-xbzv7" (OuterVolumeSpecName: "kube-api-access-xbzv7") pod "177baa81-038e-4a6d-89c4-cdad61b27e51" (UID: "177baa81-038e-4a6d-89c4-cdad61b27e51"). InnerVolumeSpecName "kube-api-access-xbzv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:41:44.158038 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.157959 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "177baa81-038e-4a6d-89c4-cdad61b27e51" (UID: "177baa81-038e-4a6d-89c4-cdad61b27e51"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:41:44.158038 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.157994 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "177baa81-038e-4a6d-89c4-cdad61b27e51" (UID: "177baa81-038e-4a6d-89c4-cdad61b27e51"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:41:44.158038 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.158003 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "177baa81-038e-4a6d-89c4-cdad61b27e51" (UID: "177baa81-038e-4a6d-89c4-cdad61b27e51"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:41:44.255728 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.255701 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:44.255831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.255745 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:44.255831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.255758 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/177baa81-038e-4a6d-89c4-cdad61b27e51-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:44.255831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.255770 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzv7\" (UniqueName: \"kubernetes.io/projected/177baa81-038e-4a6d-89c4-cdad61b27e51-kube-api-access-xbzv7\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:44.255831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.255784 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:44.255831 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.255798 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/177baa81-038e-4a6d-89c4-cdad61b27e51-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:41:44.868180 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.868150 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" Sep 11 11:41:44.868180 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.868171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr" event={"ID":"177baa81-038e-4a6d-89c4-cdad61b27e51","Type":"ContainerDied","Data":"af4cc12c9cf6569eeed29ed6569f58943c9d81f53dcb9efdfcc8f93c947525d9"} Sep 11 11:41:44.868760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.868237 2579 scope.go:117] "RemoveContainer" containerID="67b39e47c84096d4074b8da1cae5c2b8a3c4d673cef606bfe857f31d4a207964" Sep 11 11:41:44.876845 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.876826 2579 scope.go:117] "RemoveContainer" containerID="d5189f10ffa923de2211094c999f2869af8c0f8de1bd0994fb077512f5a0ef5e" Sep 11 11:41:44.884094 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.884075 2579 scope.go:117] "RemoveContainer" containerID="189f6a124d4237e541c0bfff4fb1c96a7441fdc17dbcd6e17981743f3476298c" Sep 11 11:41:44.891457 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.891432 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr"] Sep 11 11:41:44.892254 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.892233 2579 scope.go:117] "RemoveContainer" containerID="d88ed211e7017b186d40b4f8880e773ebbf7d7b7263cd9464f526391e3a21add" Sep 11 11:41:44.896235 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.896215 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-84664f6fc9-hhlvr"] Sep 11 11:41:44.899870 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.899852 2579 scope.go:117] "RemoveContainer" containerID="27ef693f421139c5e30f06f0c8e9d69924218100c21f46e6adf21b2a3cd8db33" Sep 11 11:41:44.908342 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:44.908314 2579 scope.go:117] "RemoveContainer" containerID="3f7aa227dfc982df91a94c2b46c3b5dac0e82f530ba44b8d5950ff92ff04ebf2" Sep 11 11:41:45.899106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:45.899065 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" path="/var/lib/kubelet/pods/177baa81-038e-4a6d-89c4-cdad61b27e51/volumes" Sep 11 11:41:51.577517 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577485 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-b6vts"] Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577765 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="feast-apply" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577776 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="feast-apply" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577786 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="registry" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577791 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="registry" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577797 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577802 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577809 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="feast-init" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577814 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="feast-init" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577822 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="offline" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577826 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="offline" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577832 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="offline" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577837 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="offline" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577844 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="feast-init" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577849 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="feast-init" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577854 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577859 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577868 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577873 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577879 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="feast-apply" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577884 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="feast-apply" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577891 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" Sep 11 11:41:51.577904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577896 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" Sep 11 11:41:51.578576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577951 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="ui" Sep 11 11:41:51.578576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577958 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="online" Sep 11 11:41:51.578576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577965 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="offline" Sep 11 11:41:51.578576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577970 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="registry" Sep 11 11:41:51.578576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577975 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="177baa81-038e-4a6d-89c4-cdad61b27e51" containerName="offline" Sep 11 11:41:51.578576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577982 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="online" Sep 11 11:41:51.578576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.577986 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ea27f46-75a5-4a2f-a2fe-2f846db44b81" containerName="ui" Sep 11 11:41:51.580828 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.580804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.583677 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.583652 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 11:41:51.585406 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.585384 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 11:41:51.585519 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.585466 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-42w9l\"" Sep 11 11:41:51.585599 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.585573 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 11 11:41:51.592482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.592459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b6vts"] Sep 11 11:41:51.601626 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.601603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r687b\" (UniqueName: \"kubernetes.io/projected/484958fe-a42b-4865-8c77-d67c598fbbd8-kube-api-access-r687b\") pod \"postgres-864846d4f5-b6vts\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.601769 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.601641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/484958fe-a42b-4865-8c77-d67c598fbbd8-postgresdata\") pod \"postgres-864846d4f5-b6vts\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.613760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.613736 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-c9xpb"] Sep 11 11:41:51.617251 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.617229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-c9xpb" Sep 11 11:41:51.623813 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.623784 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-c9xpb"] Sep 11 11:41:51.703020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.702997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjpk\" (UniqueName: \"kubernetes.io/projected/89b92cd8-8f08-4e7e-afeb-e454a62ba6e5-kube-api-access-rtjpk\") pod \"redis-69bbd798b6-c9xpb\" (UID: \"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5\") " pod="test-ns-feast/redis-69bbd798b6-c9xpb" Sep 11 11:41:51.703128 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.703049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r687b\" (UniqueName: \"kubernetes.io/projected/484958fe-a42b-4865-8c77-d67c598fbbd8-kube-api-access-r687b\") pod \"postgres-864846d4f5-b6vts\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.703128 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.703081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/484958fe-a42b-4865-8c77-d67c598fbbd8-postgresdata\") pod \"postgres-864846d4f5-b6vts\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.703382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.703366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/484958fe-a42b-4865-8c77-d67c598fbbd8-postgresdata\") pod \"postgres-864846d4f5-b6vts\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.711354 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.711335 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r687b\" (UniqueName: \"kubernetes.io/projected/484958fe-a42b-4865-8c77-d67c598fbbd8-kube-api-access-r687b\") pod \"postgres-864846d4f5-b6vts\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.804315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.804295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjpk\" (UniqueName: \"kubernetes.io/projected/89b92cd8-8f08-4e7e-afeb-e454a62ba6e5-kube-api-access-rtjpk\") pod \"redis-69bbd798b6-c9xpb\" (UID: \"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5\") " pod="test-ns-feast/redis-69bbd798b6-c9xpb" Sep 11 11:41:51.813996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.813973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjpk\" (UniqueName: \"kubernetes.io/projected/89b92cd8-8f08-4e7e-afeb-e454a62ba6e5-kube-api-access-rtjpk\") pod \"redis-69bbd798b6-c9xpb\" (UID: \"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5\") " pod="test-ns-feast/redis-69bbd798b6-c9xpb" Sep 11 11:41:51.895811 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.895786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:41:51.927321 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:51.927292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-c9xpb" Sep 11 11:41:52.163108 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:52.163083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b6vts"] Sep 11 11:41:52.165319 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:41:52.165284 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod484958fe_a42b_4865_8c77_d67c598fbbd8.slice/crio-e4eb28df1780da9ebfe7870d4e77812b6785c3e209571f671907780b43065146 WatchSource:0}: Error finding container e4eb28df1780da9ebfe7870d4e77812b6785c3e209571f671907780b43065146: Status 404 returned error can't find the container with id e4eb28df1780da9ebfe7870d4e77812b6785c3e209571f671907780b43065146 Sep 11 11:41:52.167014 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:52.166998 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:41:52.200203 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:52.200101 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-c9xpb"] Sep 11 11:41:52.202431 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:41:52.202410 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b92cd8_8f08_4e7e_afeb_e454a62ba6e5.slice/crio-15bf63acaae1495fba3a38b80210aca6af8a344a2e4938bd66267e0b6f0d0cda WatchSource:0}: Error finding container 15bf63acaae1495fba3a38b80210aca6af8a344a2e4938bd66267e0b6f0d0cda: Status 404 returned error can't find the container with id 15bf63acaae1495fba3a38b80210aca6af8a344a2e4938bd66267e0b6f0d0cda Sep 11 11:41:52.902322 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:52.902272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b6vts" event={"ID":"484958fe-a42b-4865-8c77-d67c598fbbd8","Type":"ContainerStarted","Data":"e4eb28df1780da9ebfe7870d4e77812b6785c3e209571f671907780b43065146"} Sep 11 11:41:52.904121 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:52.904086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-c9xpb" event={"ID":"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5","Type":"ContainerStarted","Data":"15bf63acaae1495fba3a38b80210aca6af8a344a2e4938bd66267e0b6f0d0cda"} Sep 11 11:41:54.914141 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:54.914109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-c9xpb" event={"ID":"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5","Type":"ContainerStarted","Data":"b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc"} Sep 11 11:41:54.929386 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:54.929346 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-c9xpb" podStartSLOduration=2.231140418 podStartE2EDuration="3.929330525s" totalImagesPullingTime="1.698190107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:41:54.928745142 +0000 UTC m=+691.464652140" watchObservedRunningTime="2026-09-11 11:41:54.929330525 +0000 UTC m=+691.465237521" Sep 11 11:41:56.924434 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:56.924396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b6vts" event={"ID":"484958fe-a42b-4865-8c77-d67c598fbbd8","Type":"ContainerStarted","Data":"4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6"} Sep 11 11:41:56.941064 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:41:56.941021 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-b6vts" podStartSLOduration=1.545419013 podStartE2EDuration="5.941000125s" totalImagesPullingTime="4.395581112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:41:56.939681719 +0000 UTC m=+693.475588713" watchObservedRunningTime="2026-09-11 11:41:56.941000125 +0000 UTC m=+693.476907123" Sep 11 11:42:00.408226 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.408190 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-55886d586d-sssgh"] Sep 11 11:42:00.414386 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.414368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.417477 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.417447 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 11 11:42:00.417612 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.417479 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-jj9ss\"" Sep 11 11:42:00.417612 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.417480 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 11 11:42:00.417733 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.417704 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 11 11:42:00.417780 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.417747 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 11 11:42:00.424588 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.424566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-55886d586d-sssgh"] Sep 11 11:42:00.479072 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.479041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-offline-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.479198 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.479087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-online-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.479198 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.479123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.479198 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.479152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgq9\" (UniqueName: \"kubernetes.io/projected/82f69185-f2c6-4553-96b0-05bd49af5e44-kube-api-access-gzgq9\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.479198 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.479177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/82f69185-f2c6-4553-96b0-05bd49af5e44-feast-data\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.579641 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.579617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-online-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.579805 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.579663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.579805 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.579685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgq9\" (UniqueName: \"kubernetes.io/projected/82f69185-f2c6-4553-96b0-05bd49af5e44-kube-api-access-gzgq9\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.579805 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.579768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/82f69185-f2c6-4553-96b0-05bd49af5e44-feast-data\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.579805 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:42:00.579797 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 11 11:42:00.579805 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.579805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-offline-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.580075 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:42:00.579864 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls podName:82f69185-f2c6-4553-96b0-05bd49af5e44 nodeName:}" failed. No retries permitted until 2026-09-11 11:42:01.0798483 +0000 UTC m=+697.615755276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls") pod "feast-credit-scoring-55886d586d-sssgh" (UID: "82f69185-f2c6-4553-96b0-05bd49af5e44") : secret "feast-credit-scoring-registry-tls" not found Sep 11 11:42:00.580152 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.580136 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/82f69185-f2c6-4553-96b0-05bd49af5e44-feast-data\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.582306 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.582264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-online-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.582429 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.582409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-offline-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:00.589268 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:00.589247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgq9\" (UniqueName: \"kubernetes.io/projected/82f69185-f2c6-4553-96b0-05bd49af5e44-kube-api-access-gzgq9\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:01.082987 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:01.082954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:01.085446 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:01.085420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls\") pod \"feast-credit-scoring-55886d586d-sssgh\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:01.325212 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:01.325171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:01.604382 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:01.604358 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-55886d586d-sssgh"] Sep 11 11:42:01.607148 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:42:01.607121 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f69185_f2c6_4553_96b0_05bd49af5e44.slice/crio-7b08d1772b26dccf1cbe1ef39af3cad00b437c1fbde1770d126a9e2e353d02a8 WatchSource:0}: Error finding container 7b08d1772b26dccf1cbe1ef39af3cad00b437c1fbde1770d126a9e2e353d02a8: Status 404 returned error can't find the container with id 7b08d1772b26dccf1cbe1ef39af3cad00b437c1fbde1770d126a9e2e353d02a8 Sep 11 11:42:01.944188 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:01.944151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerStarted","Data":"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac"} Sep 11 11:42:01.944350 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:01.944201 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerStarted","Data":"7b08d1772b26dccf1cbe1ef39af3cad00b437c1fbde1770d126a9e2e353d02a8"} Sep 11 11:42:02.949331 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:02.949301 2579 generic.go:356] "Generic (PLEG): container finished" podID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerID="2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac" exitCode=0 Sep 11 11:42:02.949709 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:02.949345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerDied","Data":"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac"} Sep 11 11:42:03.954615 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:03.954577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerStarted","Data":"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40"} Sep 11 11:42:09.980701 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:09.980668 2579 generic.go:356] "Generic (PLEG): container finished" podID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerID="69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40" exitCode=0 Sep 11 11:42:09.981077 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:09.980711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerDied","Data":"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40"} Sep 11 11:42:10.988325 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:10.988286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerStarted","Data":"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6"} Sep 11 11:42:10.988712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:10.988369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerStarted","Data":"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198"} Sep 11 11:42:10.988712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:10.988400 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerStarted","Data":"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53"} Sep 11 11:42:11.015020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:11.014957 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podStartSLOduration=11.014935528 podStartE2EDuration="11.014935528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:42:11.013481968 +0000 UTC m=+707.549388981" watchObservedRunningTime="2026-09-11 11:42:11.014935528 +0000 UTC m=+707.550842529" Sep 11 11:42:11.325790 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:11.325685 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:11.325790 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:11.325778 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:11.325972 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:11.325804 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:13.325556 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:13.325518 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:13.326176 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:13.326152 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:13.326292 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:13.326192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:13.327411 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:13.327377 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Sep 11 11:42:13.327411 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:13.327397 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Sep 11 11:42:13.327543 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:13.327418 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 11 11:42:16.326530 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:16.326420 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Sep 11 11:42:16.326530 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:16.326411 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Sep 11 11:42:16.326530 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:16.326425 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 11 11:42:19.326436 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:19.326395 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Sep 11 11:42:19.327124 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:19.327100 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:19.327867 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:19.327847 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:19.334655 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:19.334636 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:20.043664 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:20.043621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:22.325591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:22.325540 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.29:6571: connect: connection refused" Sep 11 11:42:25.325953 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:25.325923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:42:25.326683 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:42:25.326663 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:43:28.594419 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.594389 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-55886d586d-sssgh"] Sep 11 11:43:28.594982 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.594619 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" containerID="cri-o://dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53" gracePeriod=30 Sep 11 11:43:28.594982 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.594684 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="offline" containerID="cri-o://6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6" gracePeriod=30 Sep 11 11:43:28.594982 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.594684 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" containerID="cri-o://38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198" gracePeriod=30 Sep 11 11:43:28.613366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.613337 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b6vts"] Sep 11 11:43:28.613498 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.613480 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-b6vts" podUID="484958fe-a42b-4865-8c77-d67c598fbbd8" containerName="postgres" containerID="cri-o://4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6" gracePeriod=30 Sep 11 11:43:28.623679 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.623657 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-c9xpb"] Sep 11 11:43:28.623850 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.623832 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-c9xpb" podUID="89b92cd8-8f08-4e7e-afeb-e454a62ba6e5" containerName="redis" containerID="cri-o://b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc" gracePeriod=30 Sep 11 11:43:28.872388 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.872365 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:43:28.897429 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.897404 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-c9xpb" Sep 11 11:43:28.991881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.991851 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r687b\" (UniqueName: \"kubernetes.io/projected/484958fe-a42b-4865-8c77-d67c598fbbd8-kube-api-access-r687b\") pod \"484958fe-a42b-4865-8c77-d67c598fbbd8\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " Sep 11 11:43:28.992071 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.991913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtjpk\" (UniqueName: \"kubernetes.io/projected/89b92cd8-8f08-4e7e-afeb-e454a62ba6e5-kube-api-access-rtjpk\") pod \"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5\" (UID: \"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5\") " Sep 11 11:43:28.992071 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.991938 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/484958fe-a42b-4865-8c77-d67c598fbbd8-postgresdata\") pod \"484958fe-a42b-4865-8c77-d67c598fbbd8\" (UID: \"484958fe-a42b-4865-8c77-d67c598fbbd8\") " Sep 11 11:43:28.994117 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.994087 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b92cd8-8f08-4e7e-afeb-e454a62ba6e5-kube-api-access-rtjpk" (OuterVolumeSpecName: "kube-api-access-rtjpk") pod "89b92cd8-8f08-4e7e-afeb-e454a62ba6e5" (UID: "89b92cd8-8f08-4e7e-afeb-e454a62ba6e5"). InnerVolumeSpecName "kube-api-access-rtjpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:43:28.994227 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:28.994166 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484958fe-a42b-4865-8c77-d67c598fbbd8-kube-api-access-r687b" (OuterVolumeSpecName: "kube-api-access-r687b") pod "484958fe-a42b-4865-8c77-d67c598fbbd8" (UID: "484958fe-a42b-4865-8c77-d67c598fbbd8"). InnerVolumeSpecName "kube-api-access-r687b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:43:29.027338 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.027307 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484958fe-a42b-4865-8c77-d67c598fbbd8-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "484958fe-a42b-4865-8c77-d67c598fbbd8" (UID: "484958fe-a42b-4865-8c77-d67c598fbbd8"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:43:29.092935 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.092901 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r687b\" (UniqueName: \"kubernetes.io/projected/484958fe-a42b-4865-8c77-d67c598fbbd8-kube-api-access-r687b\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:29.092935 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.092928 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtjpk\" (UniqueName: \"kubernetes.io/projected/89b92cd8-8f08-4e7e-afeb-e454a62ba6e5-kube-api-access-rtjpk\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:29.092935 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.092939 2579 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/484958fe-a42b-4865-8c77-d67c598fbbd8-postgresdata\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:29.323681 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.323589 2579 generic.go:356] "Generic (PLEG): container finished" podID="89b92cd8-8f08-4e7e-afeb-e454a62ba6e5" containerID="b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc" exitCode=0 Sep 11 11:43:29.323681 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.323628 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-c9xpb" Sep 11 11:43:29.323944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.323674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-c9xpb" event={"ID":"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5","Type":"ContainerDied","Data":"b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc"} Sep 11 11:43:29.323944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.323744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-c9xpb" event={"ID":"89b92cd8-8f08-4e7e-afeb-e454a62ba6e5","Type":"ContainerDied","Data":"15bf63acaae1495fba3a38b80210aca6af8a344a2e4938bd66267e0b6f0d0cda"} Sep 11 11:43:29.323944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.323766 2579 scope.go:117] "RemoveContainer" containerID="b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc" Sep 11 11:43:29.325248 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.325214 2579 generic.go:356] "Generic (PLEG): container finished" podID="484958fe-a42b-4865-8c77-d67c598fbbd8" containerID="4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6" exitCode=0 Sep 11 11:43:29.325374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.325264 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-b6vts" Sep 11 11:43:29.325374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.325279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b6vts" event={"ID":"484958fe-a42b-4865-8c77-d67c598fbbd8","Type":"ContainerDied","Data":"4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6"} Sep 11 11:43:29.325374 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.325311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-b6vts" event={"ID":"484958fe-a42b-4865-8c77-d67c598fbbd8","Type":"ContainerDied","Data":"e4eb28df1780da9ebfe7870d4e77812b6785c3e209571f671907780b43065146"} Sep 11 11:43:29.332075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.332059 2579 scope.go:117] "RemoveContainer" containerID="b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc" Sep 11 11:43:29.332384 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:43:29.332356 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc\": container with ID starting with b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc not found: ID does not exist" containerID="b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc" Sep 11 11:43:29.332508 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.332392 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc"} err="failed to get container status \"b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc\": rpc error: code = NotFound desc = could not find container \"b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc\": container with ID starting with b4b515eec0b576790a2a0f30f906d2315c2b253f8b696b1b6cbc7b5cd5972fdc not found: ID does not exist" Sep 11 11:43:29.332508 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.332413 2579 scope.go:117] "RemoveContainer" containerID="4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6" Sep 11 11:43:29.340129 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.340114 2579 scope.go:117] "RemoveContainer" containerID="4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6" Sep 11 11:43:29.340377 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:43:29.340361 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6\": container with ID starting with 4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6 not found: ID does not exist" containerID="4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6" Sep 11 11:43:29.340439 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.340386 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6"} err="failed to get container status \"4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6\": rpc error: code = NotFound desc = could not find container \"4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6\": container with ID starting with 4082c31f7a87d51e4819004f1d75f93b20064ed225f5183f701ad08c52a789e6 not found: ID does not exist" Sep 11 11:43:29.349500 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.349480 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-c9xpb"] Sep 11 11:43:29.354516 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.354493 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-c9xpb"] Sep 11 11:43:29.366680 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.366662 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b6vts"] Sep 11 11:43:29.368840 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.368823 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-b6vts"] Sep 11 11:43:29.903001 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.902974 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484958fe-a42b-4865-8c77-d67c598fbbd8" path="/var/lib/kubelet/pods/484958fe-a42b-4865-8c77-d67c598fbbd8/volumes" Sep 11 11:43:29.903338 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:29.903251 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b92cd8-8f08-4e7e-afeb-e454a62ba6e5" path="/var/lib/kubelet/pods/89b92cd8-8f08-4e7e-afeb-e454a62ba6e5/volumes" Sep 11 11:43:30.040777 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:30.040679 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 11 11:43:30.332240 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:30.332163 2579 generic.go:356] "Generic (PLEG): container finished" podID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerID="38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198" exitCode=0 Sep 11 11:43:30.332376 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:30.332237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerDied","Data":"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198"} Sep 11 11:43:40.040552 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:40.040507 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 11 11:43:50.040492 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:50.040410 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 11 11:43:50.040844 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:50.040538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:43:59.248386 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.248361 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:43:59.306136 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.306105 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-offline-tls\") pod \"82f69185-f2c6-4553-96b0-05bd49af5e44\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " Sep 11 11:43:59.306136 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.306141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/82f69185-f2c6-4553-96b0-05bd49af5e44-feast-data\") pod \"82f69185-f2c6-4553-96b0-05bd49af5e44\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " Sep 11 11:43:59.306292 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.306174 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzgq9\" (UniqueName: \"kubernetes.io/projected/82f69185-f2c6-4553-96b0-05bd49af5e44-kube-api-access-gzgq9\") pod \"82f69185-f2c6-4553-96b0-05bd49af5e44\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " Sep 11 11:43:59.306292 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.306249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-online-tls\") pod \"82f69185-f2c6-4553-96b0-05bd49af5e44\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " Sep 11 11:43:59.306292 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.306273 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls\") pod \"82f69185-f2c6-4553-96b0-05bd49af5e44\" (UID: \"82f69185-f2c6-4553-96b0-05bd49af5e44\") " Sep 11 11:43:59.308361 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.308322 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-online-tls" (OuterVolumeSpecName: "online-tls") pod "82f69185-f2c6-4553-96b0-05bd49af5e44" (UID: "82f69185-f2c6-4553-96b0-05bd49af5e44"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:43:59.308481 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.308380 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "82f69185-f2c6-4553-96b0-05bd49af5e44" (UID: "82f69185-f2c6-4553-96b0-05bd49af5e44"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:43:59.308481 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.308391 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "82f69185-f2c6-4553-96b0-05bd49af5e44" (UID: "82f69185-f2c6-4553-96b0-05bd49af5e44"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:43:59.308588 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.308568 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f69185-f2c6-4553-96b0-05bd49af5e44-kube-api-access-gzgq9" (OuterVolumeSpecName: "kube-api-access-gzgq9") pod "82f69185-f2c6-4553-96b0-05bd49af5e44" (UID: "82f69185-f2c6-4553-96b0-05bd49af5e44"). InnerVolumeSpecName "kube-api-access-gzgq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:43:59.315515 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.315491 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f69185-f2c6-4553-96b0-05bd49af5e44-feast-data" (OuterVolumeSpecName: "feast-data") pod "82f69185-f2c6-4553-96b0-05bd49af5e44" (UID: "82f69185-f2c6-4553-96b0-05bd49af5e44"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:43:59.407420 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.407372 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:59.407420 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.407390 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:59.407420 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.407398 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/82f69185-f2c6-4553-96b0-05bd49af5e44-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:59.407420 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.407406 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/82f69185-f2c6-4553-96b0-05bd49af5e44-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:59.407420 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.407414 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgq9\" (UniqueName: \"kubernetes.io/projected/82f69185-f2c6-4553-96b0-05bd49af5e44-kube-api-access-gzgq9\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:43:59.446764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.446738 2579 generic.go:356] "Generic (PLEG): container finished" podID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerID="6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6" exitCode=137 Sep 11 11:43:59.446764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.446763 2579 generic.go:356] "Generic (PLEG): container finished" podID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerID="dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53" exitCode=137 Sep 11 11:43:59.446944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.446801 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" Sep 11 11:43:59.446944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.446821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerDied","Data":"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6"} Sep 11 11:43:59.446944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.446921 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerDied","Data":"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53"} Sep 11 11:43:59.446944 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.446941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-55886d586d-sssgh" event={"ID":"82f69185-f2c6-4553-96b0-05bd49af5e44","Type":"ContainerDied","Data":"7b08d1772b26dccf1cbe1ef39af3cad00b437c1fbde1770d126a9e2e353d02a8"} Sep 11 11:43:59.447072 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.446965 2579 scope.go:117] "RemoveContainer" containerID="6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6" Sep 11 11:43:59.455349 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.455331 2579 scope.go:117] "RemoveContainer" containerID="38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198" Sep 11 11:43:59.462484 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.462464 2579 scope.go:117] "RemoveContainer" containerID="dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53" Sep 11 11:43:59.470439 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.470410 2579 scope.go:117] "RemoveContainer" containerID="69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40" Sep 11 11:43:59.470662 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.470641 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-55886d586d-sssgh"] Sep 11 11:43:59.475841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.475820 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-55886d586d-sssgh"] Sep 11 11:43:59.478200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.478184 2579 scope.go:117] "RemoveContainer" containerID="2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac" Sep 11 11:43:59.485040 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.485025 2579 scope.go:117] "RemoveContainer" containerID="6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6" Sep 11 11:43:59.485312 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:43:59.485294 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6\": container with ID starting with 6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6 not found: ID does not exist" containerID="6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6" Sep 11 11:43:59.485385 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.485319 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6"} err="failed to get container status \"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6\": rpc error: code = NotFound desc = could not find container \"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6\": container with ID starting with 6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6 not found: ID does not exist" Sep 11 11:43:59.485385 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.485337 2579 scope.go:117] "RemoveContainer" containerID="38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198" Sep 11 11:43:59.485583 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:43:59.485565 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198\": container with ID starting with 38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198 not found: ID does not exist" containerID="38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198" Sep 11 11:43:59.485629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.485589 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198"} err="failed to get container status \"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198\": rpc error: code = NotFound desc = could not find container \"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198\": container with ID starting with 38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198 not found: ID does not exist" Sep 11 11:43:59.485629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.485604 2579 scope.go:117] "RemoveContainer" containerID="dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53" Sep 11 11:43:59.485943 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:43:59.485925 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53\": container with ID starting with dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53 not found: ID does not exist" containerID="dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53" Sep 11 11:43:59.486020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.485951 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53"} err="failed to get container status \"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53\": rpc error: code = NotFound desc = could not find container \"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53\": container with ID starting with dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53 not found: ID does not exist" Sep 11 11:43:59.486020 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.485970 2579 scope.go:117] "RemoveContainer" containerID="69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40" Sep 11 11:43:59.486237 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:43:59.486218 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40\": container with ID starting with 69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40 not found: ID does not exist" containerID="69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40" Sep 11 11:43:59.486283 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.486242 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40"} err="failed to get container status \"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40\": rpc error: code = NotFound desc = could not find container \"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40\": container with ID starting with 69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40 not found: ID does not exist" Sep 11 11:43:59.486283 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.486262 2579 scope.go:117] "RemoveContainer" containerID="2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac" Sep 11 11:43:59.486497 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:43:59.486482 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac\": container with ID starting with 2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac not found: ID does not exist" containerID="2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac" Sep 11 11:43:59.486558 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.486503 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac"} err="failed to get container status \"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac\": rpc error: code = NotFound desc = could not find container \"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac\": container with ID starting with 2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac not found: ID does not exist" Sep 11 11:43:59.486558 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.486523 2579 scope.go:117] "RemoveContainer" containerID="6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6" Sep 11 11:43:59.486810 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.486781 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6"} err="failed to get container status \"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6\": rpc error: code = NotFound desc = could not find container \"6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6\": container with ID starting with 6b96bfcaf054d44576b0d572607f54103bb680f66e0ae1c1ef2ffd4c79ed51d6 not found: ID does not exist" Sep 11 11:43:59.486862 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.486810 2579 scope.go:117] "RemoveContainer" containerID="38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198" Sep 11 11:43:59.487016 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.487000 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198"} err="failed to get container status \"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198\": rpc error: code = NotFound desc = could not find container \"38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198\": container with ID starting with 38f64c39cd51e550a4c9b9e034622dc6e5d57a9944f139cf07c2b3fce199b198 not found: ID does not exist" Sep 11 11:43:59.487093 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.487018 2579 scope.go:117] "RemoveContainer" containerID="dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53" Sep 11 11:43:59.487259 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.487244 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53"} err="failed to get container status \"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53\": rpc error: code = NotFound desc = could not find container \"dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53\": container with ID starting with dd516f0b9621cd9f903713956ac1d6fc96849c5e08d6fbfdee2441a431d33d53 not found: ID does not exist" Sep 11 11:43:59.487298 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.487260 2579 scope.go:117] "RemoveContainer" containerID="69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40" Sep 11 11:43:59.487489 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.487472 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40"} err="failed to get container status \"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40\": rpc error: code = NotFound desc = could not find container \"69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40\": container with ID starting with 69b65367f40524fecd1de4912a9d723878ebfecbc57121544083cefc5d11fa40 not found: ID does not exist" Sep 11 11:43:59.487529 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.487493 2579 scope.go:117] "RemoveContainer" containerID="2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac" Sep 11 11:43:59.487739 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.487698 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac"} err="failed to get container status \"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac\": rpc error: code = NotFound desc = could not find container \"2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac\": container with ID starting with 2707e3497a5d6c9d34e252a0ae1229dbb48162cc8983e3b47530bd828f1938ac not found: ID does not exist" Sep 11 11:43:59.900108 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:43:59.900081 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" path="/var/lib/kubelet/pods/82f69185-f2c6-4553-96b0-05bd49af5e44/volumes" Sep 11 11:45:56.945601 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945569 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-glwvf"] Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945842 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="feast-init" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945854 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="feast-init" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945863 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b92cd8-8f08-4e7e-afeb-e454a62ba6e5" containerName="redis" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945868 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b92cd8-8f08-4e7e-afeb-e454a62ba6e5" containerName="redis" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945878 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945883 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945888 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="offline" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945893 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="offline" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945902 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484958fe-a42b-4865-8c77-d67c598fbbd8" containerName="postgres" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945907 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="484958fe-a42b-4865-8c77-d67c598fbbd8" containerName="postgres" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945915 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="feast-apply" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945920 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="feast-apply" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945926 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945931 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945990 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b92cd8-8f08-4e7e-afeb-e454a62ba6e5" containerName="redis" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.945997 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="online" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.946003 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="484958fe-a42b-4865-8c77-d67c598fbbd8" containerName="postgres" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.946007 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="offline" Sep 11 11:45:56.946073 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.946014 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f69185-f2c6-4553-96b0-05bd49af5e44" containerName="registry" Sep 11 11:45:56.948775 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.948746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" Sep 11 11:45:56.951838 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.951817 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 11 11:45:56.953037 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.953018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-qkgt4\"" Sep 11 11:45:56.953155 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.953016 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 11 11:45:56.967226 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:56.967202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-glwvf"] Sep 11 11:45:57.049542 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.049514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rqvm\" (UniqueName: \"kubernetes.io/projected/ec8a7817-ea9d-4698-85c4-157f304a72f8-kube-api-access-9rqvm\") pod \"redis-69bbd798b6-glwvf\" (UID: \"ec8a7817-ea9d-4698-85c4-157f304a72f8\") " pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" Sep 11 11:45:57.150500 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.150474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rqvm\" (UniqueName: \"kubernetes.io/projected/ec8a7817-ea9d-4698-85c4-157f304a72f8-kube-api-access-9rqvm\") pod \"redis-69bbd798b6-glwvf\" (UID: \"ec8a7817-ea9d-4698-85c4-157f304a72f8\") " pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" Sep 11 11:45:57.160199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.160171 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rqvm\" (UniqueName: \"kubernetes.io/projected/ec8a7817-ea9d-4698-85c4-157f304a72f8-kube-api-access-9rqvm\") pod \"redis-69bbd798b6-glwvf\" (UID: \"ec8a7817-ea9d-4698-85c4-157f304a72f8\") " pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" Sep 11 11:45:57.259449 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.259395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" Sep 11 11:45:57.524959 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.524891 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-glwvf"] Sep 11 11:45:57.528079 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:45:57.528055 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec8a7817_ea9d_4698_85c4_157f304a72f8.slice/crio-54b33a7f2a3e75828900e37a0dd53b56bdca719b3e00a52db1804d74d872891d WatchSource:0}: Error finding container 54b33a7f2a3e75828900e37a0dd53b56bdca719b3e00a52db1804d74d872891d: Status 404 returned error can't find the container with id 54b33a7f2a3e75828900e37a0dd53b56bdca719b3e00a52db1804d74d872891d Sep 11 11:45:57.872216 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.872177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" event={"ID":"ec8a7817-ea9d-4698-85c4-157f304a72f8","Type":"ContainerStarted","Data":"06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5"} Sep 11 11:45:57.872413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.872225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" event={"ID":"ec8a7817-ea9d-4698-85c4-157f304a72f8","Type":"ContainerStarted","Data":"54b33a7f2a3e75828900e37a0dd53b56bdca719b3e00a52db1804d74d872891d"} Sep 11 11:45:57.888456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:45:57.888399 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" podStartSLOduration=1.888378747 podStartE2EDuration="1.888378747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:45:57.887702385 +0000 UTC m=+934.423609378" watchObservedRunningTime="2026-09-11 11:45:57.888378747 +0000 UTC m=+934.424285747" Sep 11 11:46:02.381651 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.381620 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kwzh9"] Sep 11 11:46:02.384950 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.384929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.387951 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.387923 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 11 11:46:02.393623 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.393597 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kwzh9"] Sep 11 11:46:02.488392 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.488360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpbv\" (UniqueName: \"kubernetes.io/projected/6c814ac5-99b6-493b-bedf-34a4e141e1a8-kube-api-access-2zpbv\") pod \"postgres-864846d4f5-kwzh9\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.488555 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.488416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6c814ac5-99b6-493b-bedf-34a4e141e1a8-postgresdata\") pod \"postgres-864846d4f5-kwzh9\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.589766 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.589736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpbv\" (UniqueName: \"kubernetes.io/projected/6c814ac5-99b6-493b-bedf-34a4e141e1a8-kube-api-access-2zpbv\") pod \"postgres-864846d4f5-kwzh9\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.589900 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.589792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6c814ac5-99b6-493b-bedf-34a4e141e1a8-postgresdata\") pod \"postgres-864846d4f5-kwzh9\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.590106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.590090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6c814ac5-99b6-493b-bedf-34a4e141e1a8-postgresdata\") pod \"postgres-864846d4f5-kwzh9\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.600697 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.600664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpbv\" (UniqueName: \"kubernetes.io/projected/6c814ac5-99b6-493b-bedf-34a4e141e1a8-kube-api-access-2zpbv\") pod \"postgres-864846d4f5-kwzh9\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.694818 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.694752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:46:02.966241 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:02.966217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kwzh9"] Sep 11 11:46:02.968260 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:46:02.968230 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c814ac5_99b6_493b_bedf_34a4e141e1a8.slice/crio-a6b6aa947ae4ea58171ec92cd5de6007c774b9083ff19e7931c30676b15a60dd WatchSource:0}: Error finding container a6b6aa947ae4ea58171ec92cd5de6007c774b9083ff19e7931c30676b15a60dd: Status 404 returned error can't find the container with id a6b6aa947ae4ea58171ec92cd5de6007c774b9083ff19e7931c30676b15a60dd Sep 11 11:46:03.899701 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:03.899667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" event={"ID":"6c814ac5-99b6-493b-bedf-34a4e141e1a8","Type":"ContainerStarted","Data":"29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43"} Sep 11 11:46:03.899701 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:03.899707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" event={"ID":"6c814ac5-99b6-493b-bedf-34a4e141e1a8","Type":"ContainerStarted","Data":"a6b6aa947ae4ea58171ec92cd5de6007c774b9083ff19e7931c30676b15a60dd"} Sep 11 11:46:03.914336 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:03.914288 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" podStartSLOduration=1.9142711650000002 podStartE2EDuration="1.914271165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:46:03.912797605 +0000 UTC m=+940.448704620" watchObservedRunningTime="2026-09-11 11:46:03.914271165 +0000 UTC m=+940.450178161" Sep 11 11:46:07.998445 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:07.998417 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45"] Sep 11 11:46:08.001859 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.001827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.004740 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.004704 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 11 11:46:08.004898 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.004880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-x2fwx\"" Sep 11 11:46:08.004898 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.004890 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 11 11:46:08.005408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.005389 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 11 11:46:08.013518 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.013497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45"] Sep 11 11:46:08.131261 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.131238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.131375 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.131265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-online-tls\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.131375 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.131281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fef126a5-dc33-4bd0-8382-14e6aabb1385-feast-data\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.131375 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.131298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkmq\" (UniqueName: \"kubernetes.io/projected/fef126a5-dc33-4bd0-8382-14e6aabb1385-kube-api-access-8mkmq\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.232494 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.232462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.232494 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.232500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-online-tls\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.232661 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.232516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fef126a5-dc33-4bd0-8382-14e6aabb1385-feast-data\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.232661 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.232534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkmq\" (UniqueName: \"kubernetes.io/projected/fef126a5-dc33-4bd0-8382-14e6aabb1385-kube-api-access-8mkmq\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.232661 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:46:08.232605 2579 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 11 11:46:08.232814 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:46:08.232672 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls podName:fef126a5-dc33-4bd0-8382-14e6aabb1385 nodeName:}" failed. No retries permitted until 2026-09-11 11:46:08.732654829 +0000 UTC m=+945.268561804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls") pod "feast-credit-scoring-ff58f965c-cfj45" (UID: "fef126a5-dc33-4bd0-8382-14e6aabb1385") : secret "feast-credit-scoring-registry-tls" not found Sep 11 11:46:08.232968 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.232949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fef126a5-dc33-4bd0-8382-14e6aabb1385-feast-data\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.235131 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.235114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-online-tls\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.243421 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.243401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkmq\" (UniqueName: \"kubernetes.io/projected/fef126a5-dc33-4bd0-8382-14e6aabb1385-kube-api-access-8mkmq\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.737290 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.737263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.739863 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.739839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls\") pod \"feast-credit-scoring-ff58f965c-cfj45\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:08.910459 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:08.910431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:09.189899 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:09.189875 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45"] Sep 11 11:46:09.191223 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:46:09.191196 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef126a5_dc33_4bd0_8382_14e6aabb1385.slice/crio-72dff363ab9c1bd0be233aca5fb794bb92b7c692b878674623eab03f34a6522e WatchSource:0}: Error finding container 72dff363ab9c1bd0be233aca5fb794bb92b7c692b878674623eab03f34a6522e: Status 404 returned error can't find the container with id 72dff363ab9c1bd0be233aca5fb794bb92b7c692b878674623eab03f34a6522e Sep 11 11:46:09.916408 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:09.916370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerStarted","Data":"aee8c91bfb0beb98ce3305995074bde1b922f3ab0648f511516f6ca9e8e16a83"} Sep 11 11:46:09.916588 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:09.916418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerStarted","Data":"72dff363ab9c1bd0be233aca5fb794bb92b7c692b878674623eab03f34a6522e"} Sep 11 11:46:10.921288 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:10.921254 2579 generic.go:356] "Generic (PLEG): container finished" podID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerID="aee8c91bfb0beb98ce3305995074bde1b922f3ab0648f511516f6ca9e8e16a83" exitCode=0 Sep 11 11:46:10.921683 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:10.921321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerDied","Data":"aee8c91bfb0beb98ce3305995074bde1b922f3ab0648f511516f6ca9e8e16a83"} Sep 11 11:46:11.926597 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:11.926566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerStarted","Data":"45e5d7977cb63b3baf071b455dea827362b68c46256923d70f6ea3a772ce6897"} Sep 11 11:46:17.951145 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:17.951110 2579 generic.go:356] "Generic (PLEG): container finished" podID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerID="45e5d7977cb63b3baf071b455dea827362b68c46256923d70f6ea3a772ce6897" exitCode=0 Sep 11 11:46:17.951499 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:17.951177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerDied","Data":"45e5d7977cb63b3baf071b455dea827362b68c46256923d70f6ea3a772ce6897"} Sep 11 11:46:18.958799 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:18.958751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerStarted","Data":"86b6934b894924f0ce2f019c8ce71badea0726f4c474e89f794e7afbea6909aa"} Sep 11 11:46:18.959321 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:18.958820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerStarted","Data":"1429ffc4896fd7ca85f9e8466e55919ef82c35dc16d0eb0fc19d42563ba797d5"} Sep 11 11:46:19.012565 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:19.012493 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podStartSLOduration=12.012472408 podStartE2EDuration="12.012472408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:46:19.010398463 +0000 UTC m=+955.546305484" watchObservedRunningTime="2026-09-11 11:46:19.012472408 +0000 UTC m=+955.548379405" Sep 11 11:46:20.911467 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:20.911425 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:20.911983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:20.911498 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:20.912878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:20.912852 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 11 11:46:20.912956 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:20.912893 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 11 11:46:23.911191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:23.911153 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 11 11:46:23.912115 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:23.912088 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 11 11:46:26.911780 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:26.911735 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 11 11:46:26.920423 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:26.920402 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:26.920864 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:26.920844 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:26.925026 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:26.925006 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:28.911497 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:28.911456 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:29.911868 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:29.911837 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:30.004585 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:30.004557 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:46:33.791337 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.791303 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl"] Sep 11 11:46:33.796181 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.796165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.799274 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.799248 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 11 11:46:33.799388 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.799311 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 11 11:46:33.799517 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.799504 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 11 11:46:33.799654 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.799630 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-mmzjp\"" Sep 11 11:46:33.806803 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.806783 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl"] Sep 11 11:46:33.844911 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.844879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.844911 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.844913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72d57e62-731b-493c-85bf-b2113df37244-feast-data\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.845056 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.844960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.845056 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.844989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7ldd\" (UniqueName: \"kubernetes.io/projected/72d57e62-731b-493c-85bf-b2113df37244-kube-api-access-k7ldd\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.945924 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.945900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.946054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.945931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ldd\" (UniqueName: \"kubernetes.io/projected/72d57e62-731b-493c-85bf-b2113df37244-kube-api-access-k7ldd\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.946054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.945980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.946054 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.946001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72d57e62-731b-493c-85bf-b2113df37244-feast-data\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.946175 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:46:33.946053 2579 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 11 11:46:33.946175 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:46:33.946118 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls podName:72d57e62-731b-493c-85bf-b2113df37244 nodeName:}" failed. No retries permitted until 2026-09-11 11:46:34.446100647 +0000 UTC m=+970.982007633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls") pod "feast-driver-ranking-bf6bf9b58-rtwkl" (UID: "72d57e62-731b-493c-85bf-b2113df37244") : secret "feast-driver-ranking-online-tls" not found Sep 11 11:46:33.946175 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:46:33.946129 2579 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 11 11:46:33.946303 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:46:33.946186 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls podName:72d57e62-731b-493c-85bf-b2113df37244 nodeName:}" failed. No retries permitted until 2026-09-11 11:46:34.446172505 +0000 UTC m=+970.982079484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls") pod "feast-driver-ranking-bf6bf9b58-rtwkl" (UID: "72d57e62-731b-493c-85bf-b2113df37244") : secret "feast-driver-ranking-registry-tls" not found Sep 11 11:46:33.946383 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.946364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72d57e62-731b-493c-85bf-b2113df37244-feast-data\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:33.955760 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:33.955736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ldd\" (UniqueName: \"kubernetes.io/projected/72d57e62-731b-493c-85bf-b2113df37244-kube-api-access-k7ldd\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:34.449450 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:34.449415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:34.449617 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:34.449489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:34.452045 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:34.452020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:34.452166 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:34.452108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls\") pod \"feast-driver-ranking-bf6bf9b58-rtwkl\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:34.704904 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:34.704819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:34.979767 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:34.979744 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl"] Sep 11 11:46:34.981472 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:46:34.981444 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d57e62_731b_493c_85bf_b2113df37244.slice/crio-caf497e3750139c5907f6428a2d590d2e83e9d5ed0694ebd7d4e39218079ec74 WatchSource:0}: Error finding container caf497e3750139c5907f6428a2d590d2e83e9d5ed0694ebd7d4e39218079ec74: Status 404 returned error can't find the container with id caf497e3750139c5907f6428a2d590d2e83e9d5ed0694ebd7d4e39218079ec74 Sep 11 11:46:35.022872 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:35.022843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerStarted","Data":"caf497e3750139c5907f6428a2d590d2e83e9d5ed0694ebd7d4e39218079ec74"} Sep 11 11:46:36.028253 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:36.028219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerStarted","Data":"0a5b53d87692b8f3a28f84aa11e4e1d8fb4887a2020cb9365c764fd68ef45135"} Sep 11 11:46:40.045604 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:40.045523 2579 generic.go:356] "Generic (PLEG): container finished" podID="72d57e62-731b-493c-85bf-b2113df37244" containerID="0a5b53d87692b8f3a28f84aa11e4e1d8fb4887a2020cb9365c764fd68ef45135" exitCode=0 Sep 11 11:46:40.045973 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:40.045598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerDied","Data":"0a5b53d87692b8f3a28f84aa11e4e1d8fb4887a2020cb9365c764fd68ef45135"} Sep 11 11:46:41.051769 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:41.051732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerStarted","Data":"87170028d2d95a45ebedd63a99788ce21a1f98fed6fac6e08d327c18eb555994"} Sep 11 11:46:47.078225 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:47.078128 2579 generic.go:356] "Generic (PLEG): container finished" podID="72d57e62-731b-493c-85bf-b2113df37244" containerID="87170028d2d95a45ebedd63a99788ce21a1f98fed6fac6e08d327c18eb555994" exitCode=0 Sep 11 11:46:47.078225 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:47.078195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerDied","Data":"87170028d2d95a45ebedd63a99788ce21a1f98fed6fac6e08d327c18eb555994"} Sep 11 11:46:48.087418 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:48.087381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerStarted","Data":"1cd43b4e646a3e70d99541a23bd025cc0a7826f784b39010845db900f979f478"} Sep 11 11:46:48.087813 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:48.087443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerStarted","Data":"809bfd48ad9e052de2a1bbc4f2533c2932b2ec1f283a52ffcb22ca71931743bc"} Sep 11 11:46:48.112634 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:48.112584 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podStartSLOduration=15.112568746 podStartE2EDuration="15.112568746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:46:48.110964361 +0000 UTC m=+984.646871359" watchObservedRunningTime="2026-09-11 11:46:48.112568746 +0000 UTC m=+984.648475744" Sep 11 11:46:49.705890 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:49.705852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:49.706284 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:49.705926 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:49.707432 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:49.707393 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Sep 11 11:46:49.707432 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:49.707418 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 11 11:46:52.705983 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:52.705927 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Sep 11 11:46:52.706389 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:52.705984 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 11 11:46:54.705712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:54.705674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:54.706103 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:54.705774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:55.705966 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:55.705915 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" Sep 11 11:46:55.712021 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:55.711993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:55.715356 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:55.715336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:58.705819 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:58.705780 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:46:58.706565 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:46:58.706535 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:48:16.571603 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:16.571518 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb"] Sep 11 11:48:16.572146 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:16.571803 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" podUID="ce174939-e51d-448e-a9c3-e8e71eb15996" containerName="manager" containerID="cri-o://af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049" gracePeriod=10 Sep 11 11:48:16.812441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:16.812413 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:48:16.944703 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:16.944678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnwp6\" (UniqueName: \"kubernetes.io/projected/ce174939-e51d-448e-a9c3-e8e71eb15996-kube-api-access-nnwp6\") pod \"ce174939-e51d-448e-a9c3-e8e71eb15996\" (UID: \"ce174939-e51d-448e-a9c3-e8e71eb15996\") " Sep 11 11:48:16.947816 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:16.947780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce174939-e51d-448e-a9c3-e8e71eb15996-kube-api-access-nnwp6" (OuterVolumeSpecName: "kube-api-access-nnwp6") pod "ce174939-e51d-448e-a9c3-e8e71eb15996" (UID: "ce174939-e51d-448e-a9c3-e8e71eb15996"). InnerVolumeSpecName "kube-api-access-nnwp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:48:17.044961 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.044932 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45"] Sep 11 11:48:17.045208 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.045162 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="registry" containerID="cri-o://1429ffc4896fd7ca85f9e8466e55919ef82c35dc16d0eb0fc19d42563ba797d5" gracePeriod=30 Sep 11 11:48:17.045293 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.045204 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwp6\" (UniqueName: \"kubernetes.io/projected/ce174939-e51d-448e-a9c3-e8e71eb15996-kube-api-access-nnwp6\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:17.045293 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.045256 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" containerID="cri-o://86b6934b894924f0ce2f019c8ce71badea0726f4c474e89f794e7afbea6909aa" gracePeriod=30 Sep 11 11:48:17.052025 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.052004 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl"] Sep 11 11:48:17.052191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.052171 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="registry" containerID="cri-o://809bfd48ad9e052de2a1bbc4f2533c2932b2ec1f283a52ffcb22ca71931743bc" gracePeriod=30 Sep 11 11:48:17.052248 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.052191 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" containerID="cri-o://1cd43b4e646a3e70d99541a23bd025cc0a7826f784b39010845db900f979f478" gracePeriod=30 Sep 11 11:48:17.445287 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.445245 2579 generic.go:356] "Generic (PLEG): container finished" podID="ce174939-e51d-448e-a9c3-e8e71eb15996" containerID="af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049" exitCode=0 Sep 11 11:48:17.445500 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.445324 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" Sep 11 11:48:17.445500 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.445345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" event={"ID":"ce174939-e51d-448e-a9c3-e8e71eb15996","Type":"ContainerDied","Data":"af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049"} Sep 11 11:48:17.445500 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.445423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb" event={"ID":"ce174939-e51d-448e-a9c3-e8e71eb15996","Type":"ContainerDied","Data":"ce94e22323fee704963fb6aa7e2bc75aaf00eda71fdcfdc66cd367ef302ea3cc"} Sep 11 11:48:17.445500 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.445447 2579 scope.go:117] "RemoveContainer" containerID="af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049" Sep 11 11:48:17.454195 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.454179 2579 scope.go:117] "RemoveContainer" containerID="af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049" Sep 11 11:48:17.454479 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:48:17.454459 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049\": container with ID starting with af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049 not found: ID does not exist" containerID="af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049" Sep 11 11:48:17.454566 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.454488 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049"} err="failed to get container status \"af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049\": rpc error: code = NotFound desc = could not find container \"af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049\": container with ID starting with af7c1886df7a57536911ab32b9eb158a807641ace35742f924eb82e346b01049 not found: ID does not exist" Sep 11 11:48:17.474623 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.474596 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb"] Sep 11 11:48:17.476193 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.476171 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-8c5f9c9d7-dhkvb"] Sep 11 11:48:17.900067 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:17.900040 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce174939-e51d-448e-a9c3-e8e71eb15996" path="/var/lib/kubelet/pods/ce174939-e51d-448e-a9c3-e8e71eb15996/volumes" Sep 11 11:48:18.453497 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:18.453466 2579 generic.go:356] "Generic (PLEG): container finished" podID="72d57e62-731b-493c-85bf-b2113df37244" containerID="1cd43b4e646a3e70d99541a23bd025cc0a7826f784b39010845db900f979f478" exitCode=0 Sep 11 11:48:18.453694 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:18.453541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerDied","Data":"1cd43b4e646a3e70d99541a23bd025cc0a7826f784b39010845db900f979f478"} Sep 11 11:48:18.456526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:18.456466 2579 generic.go:356] "Generic (PLEG): container finished" podID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerID="86b6934b894924f0ce2f019c8ce71badea0726f4c474e89f794e7afbea6909aa" exitCode=0 Sep 11 11:48:18.456643 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:18.456545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerDied","Data":"86b6934b894924f0ce2f019c8ce71badea0726f4c474e89f794e7afbea6909aa"} Sep 11 11:48:19.029310 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.029273 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kwzh9"] Sep 11 11:48:19.029786 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.029475 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" podUID="6c814ac5-99b6-493b-bedf-34a4e141e1a8" containerName="postgres" containerID="cri-o://29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43" gracePeriod=30 Sep 11 11:48:19.044482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.044453 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-glwvf"] Sep 11 11:48:19.044651 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.044631 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" podUID="ec8a7817-ea9d-4698-85c4-157f304a72f8" containerName="redis" containerID="cri-o://06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5" gracePeriod=30 Sep 11 11:48:19.294633 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.294612 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:48:19.297129 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.297111 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" Sep 11 11:48:19.363633 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.363605 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6c814ac5-99b6-493b-bedf-34a4e141e1a8-postgresdata\") pod \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " Sep 11 11:48:19.363816 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.363768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zpbv\" (UniqueName: \"kubernetes.io/projected/6c814ac5-99b6-493b-bedf-34a4e141e1a8-kube-api-access-2zpbv\") pod \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\" (UID: \"6c814ac5-99b6-493b-bedf-34a4e141e1a8\") " Sep 11 11:48:19.363816 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.363804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rqvm\" (UniqueName: \"kubernetes.io/projected/ec8a7817-ea9d-4698-85c4-157f304a72f8-kube-api-access-9rqvm\") pod \"ec8a7817-ea9d-4698-85c4-157f304a72f8\" (UID: \"ec8a7817-ea9d-4698-85c4-157f304a72f8\") " Sep 11 11:48:19.366449 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.366421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c814ac5-99b6-493b-bedf-34a4e141e1a8-kube-api-access-2zpbv" (OuterVolumeSpecName: "kube-api-access-2zpbv") pod "6c814ac5-99b6-493b-bedf-34a4e141e1a8" (UID: "6c814ac5-99b6-493b-bedf-34a4e141e1a8"). InnerVolumeSpecName "kube-api-access-2zpbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:48:19.366567 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.366521 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8a7817-ea9d-4698-85c4-157f304a72f8-kube-api-access-9rqvm" (OuterVolumeSpecName: "kube-api-access-9rqvm") pod "ec8a7817-ea9d-4698-85c4-157f304a72f8" (UID: "ec8a7817-ea9d-4698-85c4-157f304a72f8"). InnerVolumeSpecName "kube-api-access-9rqvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:48:19.394700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.394674 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c814ac5-99b6-493b-bedf-34a4e141e1a8-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "6c814ac5-99b6-493b-bedf-34a4e141e1a8" (UID: "6c814ac5-99b6-493b-bedf-34a4e141e1a8"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:19.462262 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.462233 2579 generic.go:356] "Generic (PLEG): container finished" podID="ec8a7817-ea9d-4698-85c4-157f304a72f8" containerID="06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5" exitCode=0 Sep 11 11:48:19.462373 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.462273 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" Sep 11 11:48:19.462373 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.462324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" event={"ID":"ec8a7817-ea9d-4698-85c4-157f304a72f8","Type":"ContainerDied","Data":"06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5"} Sep 11 11:48:19.462373 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.462365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-glwvf" event={"ID":"ec8a7817-ea9d-4698-85c4-157f304a72f8","Type":"ContainerDied","Data":"54b33a7f2a3e75828900e37a0dd53b56bdca719b3e00a52db1804d74d872891d"} Sep 11 11:48:19.462524 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.462383 2579 scope.go:117] "RemoveContainer" containerID="06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5" Sep 11 11:48:19.463572 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.463552 2579 generic.go:356] "Generic (PLEG): container finished" podID="6c814ac5-99b6-493b-bedf-34a4e141e1a8" containerID="29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43" exitCode=0 Sep 11 11:48:19.463674 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.463596 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" Sep 11 11:48:19.463674 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.463637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" event={"ID":"6c814ac5-99b6-493b-bedf-34a4e141e1a8","Type":"ContainerDied","Data":"29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43"} Sep 11 11:48:19.463674 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.463669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-kwzh9" event={"ID":"6c814ac5-99b6-493b-bedf-34a4e141e1a8","Type":"ContainerDied","Data":"a6b6aa947ae4ea58171ec92cd5de6007c774b9083ff19e7931c30676b15a60dd"} Sep 11 11:48:19.464339 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.464325 2579 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/6c814ac5-99b6-493b-bedf-34a4e141e1a8-postgresdata\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:19.464395 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.464345 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpbv\" (UniqueName: \"kubernetes.io/projected/6c814ac5-99b6-493b-bedf-34a4e141e1a8-kube-api-access-2zpbv\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:19.464395 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.464359 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rqvm\" (UniqueName: \"kubernetes.io/projected/ec8a7817-ea9d-4698-85c4-157f304a72f8-kube-api-access-9rqvm\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:19.471485 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.471460 2579 scope.go:117] "RemoveContainer" containerID="06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5" Sep 11 11:48:19.471767 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:48:19.471746 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5\": container with ID starting with 06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5 not found: ID does not exist" containerID="06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5" Sep 11 11:48:19.471845 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.471779 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5"} err="failed to get container status \"06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5\": rpc error: code = NotFound desc = could not find container \"06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5\": container with ID starting with 06d1669d761ddee44b3e1edc88a89856ae073694764039bd938706995f46b6b5 not found: ID does not exist" Sep 11 11:48:19.471845 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.471801 2579 scope.go:117] "RemoveContainer" containerID="29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43" Sep 11 11:48:19.479118 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.479104 2579 scope.go:117] "RemoveContainer" containerID="29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43" Sep 11 11:48:19.479369 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:48:19.479352 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43\": container with ID starting with 29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43 not found: ID does not exist" containerID="29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43" Sep 11 11:48:19.479416 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.479375 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43"} err="failed to get container status \"29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43\": rpc error: code = NotFound desc = could not find container \"29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43\": container with ID starting with 29f21cc384c4f0923eec0dc548fe92367abf966f92566eb58dfea27ce331ad43 not found: ID does not exist" Sep 11 11:48:19.489386 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.489365 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-glwvf"] Sep 11 11:48:19.496443 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.496423 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-glwvf"] Sep 11 11:48:19.520780 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.520756 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kwzh9"] Sep 11 11:48:19.524914 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.524896 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-kwzh9"] Sep 11 11:48:19.898825 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.898791 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c814ac5-99b6-493b-bedf-34a4e141e1a8" path="/var/lib/kubelet/pods/6c814ac5-99b6-493b-bedf-34a4e141e1a8/volumes" Sep 11 11:48:19.899100 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:19.899087 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8a7817-ea9d-4698-85c4-157f304a72f8" path="/var/lib/kubelet/pods/ec8a7817-ea9d-4698-85c4-157f304a72f8/volumes" Sep 11 11:48:25.712542 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:25.712500 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 11 11:48:26.921238 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:26.921197 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 11 11:48:35.712866 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:35.712818 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 11 11:48:36.920896 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:36.920856 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 11 11:48:45.712893 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:45.712850 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" Sep 11 11:48:45.713295 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:45.712994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:48:46.921428 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:46.921390 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 11 11:48:46.921896 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:46.921513 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:48:47.572063 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.572034 2579 generic.go:356] "Generic (PLEG): container finished" podID="72d57e62-731b-493c-85bf-b2113df37244" containerID="809bfd48ad9e052de2a1bbc4f2533c2932b2ec1f283a52ffcb22ca71931743bc" exitCode=137 Sep 11 11:48:47.572195 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.572117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerDied","Data":"809bfd48ad9e052de2a1bbc4f2533c2932b2ec1f283a52ffcb22ca71931743bc"} Sep 11 11:48:47.575159 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.575137 2579 generic.go:356] "Generic (PLEG): container finished" podID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerID="1429ffc4896fd7ca85f9e8466e55919ef82c35dc16d0eb0fc19d42563ba797d5" exitCode=137 Sep 11 11:48:47.575276 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.575188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerDied","Data":"1429ffc4896fd7ca85f9e8466e55919ef82c35dc16d0eb0fc19d42563ba797d5"} Sep 11 11:48:47.704609 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.704587 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:48:47.707286 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.707267 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:48:47.880302 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls\") pod \"72d57e62-731b-493c-85bf-b2113df37244\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " Sep 11 11:48:47.880453 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fef126a5-dc33-4bd0-8382-14e6aabb1385-feast-data\") pod \"fef126a5-dc33-4bd0-8382-14e6aabb1385\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " Sep 11 11:48:47.880453 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880341 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7ldd\" (UniqueName: \"kubernetes.io/projected/72d57e62-731b-493c-85bf-b2113df37244-kube-api-access-k7ldd\") pod \"72d57e62-731b-493c-85bf-b2113df37244\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " Sep 11 11:48:47.880453 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880388 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mkmq\" (UniqueName: \"kubernetes.io/projected/fef126a5-dc33-4bd0-8382-14e6aabb1385-kube-api-access-8mkmq\") pod \"fef126a5-dc33-4bd0-8382-14e6aabb1385\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " Sep 11 11:48:47.880453 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880420 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls\") pod \"72d57e62-731b-493c-85bf-b2113df37244\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " Sep 11 11:48:47.880671 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-online-tls\") pod \"fef126a5-dc33-4bd0-8382-14e6aabb1385\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " Sep 11 11:48:47.880671 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880476 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72d57e62-731b-493c-85bf-b2113df37244-feast-data\") pod \"72d57e62-731b-493c-85bf-b2113df37244\" (UID: \"72d57e62-731b-493c-85bf-b2113df37244\") " Sep 11 11:48:47.880671 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.880511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls\") pod \"fef126a5-dc33-4bd0-8382-14e6aabb1385\" (UID: \"fef126a5-dc33-4bd0-8382-14e6aabb1385\") " Sep 11 11:48:47.881482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.881408 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d57e62-731b-493c-85bf-b2113df37244-feast-data" (OuterVolumeSpecName: "feast-data") pod "72d57e62-731b-493c-85bf-b2113df37244" (UID: "72d57e62-731b-493c-85bf-b2113df37244"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:47.882793 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.882756 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "72d57e62-731b-493c-85bf-b2113df37244" (UID: "72d57e62-731b-493c-85bf-b2113df37244"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:48:47.883199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.883174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fef126a5-dc33-4bd0-8382-14e6aabb1385" (UID: "fef126a5-dc33-4bd0-8382-14e6aabb1385"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:48:47.883307 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.883285 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-online-tls" (OuterVolumeSpecName: "online-tls") pod "fef126a5-dc33-4bd0-8382-14e6aabb1385" (UID: "fef126a5-dc33-4bd0-8382-14e6aabb1385"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:48:47.883359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.883303 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls" (OuterVolumeSpecName: "online-tls") pod "72d57e62-731b-493c-85bf-b2113df37244" (UID: "72d57e62-731b-493c-85bf-b2113df37244"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:48:47.883359 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.883313 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef126a5-dc33-4bd0-8382-14e6aabb1385-kube-api-access-8mkmq" (OuterVolumeSpecName: "kube-api-access-8mkmq") pod "fef126a5-dc33-4bd0-8382-14e6aabb1385" (UID: "fef126a5-dc33-4bd0-8382-14e6aabb1385"). InnerVolumeSpecName "kube-api-access-8mkmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:48:47.883426 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.883358 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d57e62-731b-493c-85bf-b2113df37244-kube-api-access-k7ldd" (OuterVolumeSpecName: "kube-api-access-k7ldd") pod "72d57e62-731b-493c-85bf-b2113df37244" (UID: "72d57e62-731b-493c-85bf-b2113df37244"). InnerVolumeSpecName "kube-api-access-k7ldd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:48:47.890072 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.890050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef126a5-dc33-4bd0-8382-14e6aabb1385-feast-data" (OuterVolumeSpecName: "feast-data") pod "fef126a5-dc33-4bd0-8382-14e6aabb1385" (UID: "fef126a5-dc33-4bd0-8382-14e6aabb1385"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:47.981908 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981886 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7ldd\" (UniqueName: \"kubernetes.io/projected/72d57e62-731b-493c-85bf-b2113df37244-kube-api-access-k7ldd\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:47.981908 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981908 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkmq\" (UniqueName: \"kubernetes.io/projected/fef126a5-dc33-4bd0-8382-14e6aabb1385-kube-api-access-8mkmq\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:47.982200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981935 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:47.982200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981944 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:47.982200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981953 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/72d57e62-731b-493c-85bf-b2113df37244-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:47.982200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981960 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fef126a5-dc33-4bd0-8382-14e6aabb1385-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:47.982200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981966 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/72d57e62-731b-493c-85bf-b2113df37244-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:47.982200 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:47.981973 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fef126a5-dc33-4bd0-8382-14e6aabb1385-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:48:48.581287 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.581261 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" Sep 11 11:48:48.581287 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.581272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl" event={"ID":"72d57e62-731b-493c-85bf-b2113df37244","Type":"ContainerDied","Data":"caf497e3750139c5907f6428a2d590d2e83e9d5ed0694ebd7d4e39218079ec74"} Sep 11 11:48:48.581521 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.581345 2579 scope.go:117] "RemoveContainer" containerID="1cd43b4e646a3e70d99541a23bd025cc0a7826f784b39010845db900f979f478" Sep 11 11:48:48.583956 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.583932 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" Sep 11 11:48:48.584076 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.583931 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45" event={"ID":"fef126a5-dc33-4bd0-8382-14e6aabb1385","Type":"ContainerDied","Data":"72dff363ab9c1bd0be233aca5fb794bb92b7c692b878674623eab03f34a6522e"} Sep 11 11:48:48.590036 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.590016 2579 scope.go:117] "RemoveContainer" containerID="809bfd48ad9e052de2a1bbc4f2533c2932b2ec1f283a52ffcb22ca71931743bc" Sep 11 11:48:48.597998 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.597980 2579 scope.go:117] "RemoveContainer" containerID="87170028d2d95a45ebedd63a99788ce21a1f98fed6fac6e08d327c18eb555994" Sep 11 11:48:48.605943 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.605906 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45"] Sep 11 11:48:48.606143 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.606118 2579 scope.go:117] "RemoveContainer" containerID="0a5b53d87692b8f3a28f84aa11e4e1d8fb4887a2020cb9365c764fd68ef45135" Sep 11 11:48:48.612237 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.612217 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-ff58f965c-cfj45"] Sep 11 11:48:48.613366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.613355 2579 scope.go:117] "RemoveContainer" containerID="86b6934b894924f0ce2f019c8ce71badea0726f4c474e89f794e7afbea6909aa" Sep 11 11:48:48.620228 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.620213 2579 scope.go:117] "RemoveContainer" containerID="1429ffc4896fd7ca85f9e8466e55919ef82c35dc16d0eb0fc19d42563ba797d5" Sep 11 11:48:48.625091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.625073 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl"] Sep 11 11:48:48.627780 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.627766 2579 scope.go:117] "RemoveContainer" containerID="45e5d7977cb63b3baf071b455dea827362b68c46256923d70f6ea3a772ce6897" Sep 11 11:48:48.629502 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.629482 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-bf6bf9b58-rtwkl"] Sep 11 11:48:48.634672 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:48.634656 2579 scope.go:117] "RemoveContainer" containerID="aee8c91bfb0beb98ce3305995074bde1b922f3ab0648f511516f6ca9e8e16a83" Sep 11 11:48:49.899584 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:49.899551 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d57e62-731b-493c-85bf-b2113df37244" path="/var/lib/kubelet/pods/72d57e62-731b-493c-85bf-b2113df37244/volumes" Sep 11 11:48:49.900060 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:49.900046 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" path="/var/lib/kubelet/pods/fef126a5-dc33-4bd0-8382-14e6aabb1385/volumes" Sep 11 11:48:50.922437 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922404 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg"] Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922654 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922664 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922675 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="registry" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922681 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="registry" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922690 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce174939-e51d-448e-a9c3-e8e71eb15996" containerName="manager" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922695 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce174939-e51d-448e-a9c3-e8e71eb15996" containerName="manager" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922702 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="feast-apply" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922708 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="feast-apply" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922731 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8a7817-ea9d-4698-85c4-157f304a72f8" containerName="redis" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922737 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8a7817-ea9d-4698-85c4-157f304a72f8" containerName="redis" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922744 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="feast-apply" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922749 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="feast-apply" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922756 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922761 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922767 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c814ac5-99b6-493b-bedf-34a4e141e1a8" containerName="postgres" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922771 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c814ac5-99b6-493b-bedf-34a4e141e1a8" containerName="postgres" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922786 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="feast-init" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922791 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="feast-init" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922798 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="feast-init" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922802 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="feast-init" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922808 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="registry" Sep 11 11:48:50.922823 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922813 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="registry" Sep 11 11:48:50.923482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922855 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce174939-e51d-448e-a9c3-e8e71eb15996" containerName="manager" Sep 11 11:48:50.923482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922864 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8a7817-ea9d-4698-85c4-157f304a72f8" containerName="redis" Sep 11 11:48:50.923482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922871 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c814ac5-99b6-493b-bedf-34a4e141e1a8" containerName="postgres" Sep 11 11:48:50.923482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922875 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="registry" Sep 11 11:48:50.923482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922881 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="registry" Sep 11 11:48:50.923482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922886 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fef126a5-dc33-4bd0-8382-14e6aabb1385" containerName="online" Sep 11 11:48:50.923482 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.922891 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d57e62-731b-493c-85bf-b2113df37244" containerName="online" Sep 11 11:48:50.926081 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.926065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:50.931334 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.931313 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 11:48:50.932013 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.931986 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 11:48:50.932185 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.932148 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 11:48:50.932185 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.932156 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 11:48:50.932185 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.932064 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-8j7qf\"" Sep 11 11:48:50.932439 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.932036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 11:48:50.932439 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.932079 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 11:48:50.944031 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:50.944006 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg"] Sep 11 11:48:51.104993 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.104968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.105121 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.105004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.105121 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.105033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwdz\" (UniqueName: \"kubernetes.io/projected/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-kube-api-access-4mwdz\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.105199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.105115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.105199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.105156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.105199 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.105173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.205658 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.205591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwdz\" (UniqueName: \"kubernetes.io/projected/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-kube-api-access-4mwdz\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.205658 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.205627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.205869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.205664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.205869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.205768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.205869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.205808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.205869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.205826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.206189 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.206162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.208370 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.208336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.208480 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.208379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.208545 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.208508 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.208545 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.208513 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.214609 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.214590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwdz\" (UniqueName: \"kubernetes.io/projected/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-kube-api-access-4mwdz\") pod \"feast-simple-feast-setup-57576b8fcf-fj9lg\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.235630 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.235605 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:48:51.514920 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.514835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg"] Sep 11 11:48:51.517797 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:48:51.517771 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b1a0aa5_d4fb_4ce9_86d2_8522a98b6d12.slice/crio-fef48d4a01f3d1caba62b11c4e9473d211183395bf1c17b49e7f6b2b5619ff77 WatchSource:0}: Error finding container fef48d4a01f3d1caba62b11c4e9473d211183395bf1c17b49e7f6b2b5619ff77: Status 404 returned error can't find the container with id fef48d4a01f3d1caba62b11c4e9473d211183395bf1c17b49e7f6b2b5619ff77 Sep 11 11:48:51.519804 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.519787 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:48:51.598401 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:48:51.598374 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerStarted","Data":"fef48d4a01f3d1caba62b11c4e9473d211183395bf1c17b49e7f6b2b5619ff77"} Sep 11 11:49:05.665417 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:05.665373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerStarted","Data":"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634"} Sep 11 11:49:10.685622 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:10.685584 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerID="7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634" exitCode=0 Sep 11 11:49:10.686076 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:10.685661 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerDied","Data":"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634"} Sep 11 11:49:11.691366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:11.691334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerStarted","Data":"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228"} Sep 11 11:49:16.712005 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:16.711923 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerID="506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228" exitCode=0 Sep 11 11:49:16.712005 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:16.711993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerDied","Data":"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228"} Sep 11 11:49:17.722167 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:17.722114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerStarted","Data":"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476"} Sep 11 11:49:17.722628 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:17.722192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerStarted","Data":"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a"} Sep 11 11:49:17.722628 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:17.722217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerStarted","Data":"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad"} Sep 11 11:49:17.722628 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:17.722239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerStarted","Data":"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53"} Sep 11 11:49:17.751305 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:17.751226 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podStartSLOduration=14.099952598 podStartE2EDuration="27.751204576s" totalImagesPullingTime="13.651251978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:49:17.749225813 +0000 UTC m=+1134.285132822" watchObservedRunningTime="2026-09-11 11:49:17.751204576 +0000 UTC m=+1134.287111574" Sep 11 11:49:18.236021 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.235981 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:18.236226 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.236053 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:18.236226 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.236078 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:18.236226 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.236096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:18.238340 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.238185 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 11 11:49:18.238340 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.238232 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused" Sep 11 11:49:18.238340 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.238281 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused" Sep 11 11:49:18.238340 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:18.238232 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 11 11:49:21.236191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.236146 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:21.236700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.236301 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:21.236700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.236470 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:21.236700 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.236508 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:21.237045 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.237001 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused" Sep 11 11:49:21.237148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.237000 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 11 11:49:21.237148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.236999 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused" Sep 11 11:49:21.237504 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:21.237478 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 11 11:49:24.237114 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:24.236594 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 11 11:49:24.237114 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:24.236620 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused" Sep 11 11:49:24.237114 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:24.236751 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.34:8816: connect: connection refused" Sep 11 11:49:24.237114 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:24.236887 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 11 11:49:27.237881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.237853 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:27.238353 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.237991 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:27.238353 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.238315 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:27.238887 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.238864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:27.239040 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.238953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:27.239040 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.239008 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:27.243076 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.243054 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:49:27.775111 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:49:27.775082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:50:29.267077 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.267034 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg"] Sep 11 11:50:29.267615 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.267517 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="registry" containerID="cri-o://64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53" gracePeriod=30 Sep 11 11:50:29.267615 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.267570 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" containerID="cri-o://7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad" gracePeriod=30 Sep 11 11:50:29.267761 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.267570 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="offline" containerID="cri-o://45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a" gracePeriod=30 Sep 11 11:50:29.267828 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.267764 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" containerID="cri-o://c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476" gracePeriod=30 Sep 11 11:50:29.696136 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.696100 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm"] Sep 11 11:50:29.700060 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.700040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.703663 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.703643 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-rplv7\"" Sep 11 11:50:29.714291 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.714266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm"] Sep 11 11:50:29.753554 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.753519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.753750 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.753569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89hp\" (UniqueName: \"kubernetes.io/projected/6d3b2737-dada-4b82-87d5-b43ae1891805-kube-api-access-p89hp\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.753750 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.753609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3b2737-dada-4b82-87d5-b43ae1891805-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.753852 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.753754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.753852 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.753803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.753852 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.753832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.854694 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.854650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89hp\" (UniqueName: \"kubernetes.io/projected/6d3b2737-dada-4b82-87d5-b43ae1891805-kube-api-access-p89hp\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.854875 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.854746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3b2737-dada-4b82-87d5-b43ae1891805-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.854875 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.854858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.854977 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.854906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.854977 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.854931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.855078 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.854974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.855078 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:50:29.855023 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 11 11:50:29.855078 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:50:29.855033 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 11:50:29.855078 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:50:29.855073 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 11 11:50:29.855262 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:50:29.855104 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls podName:6d3b2737-dada-4b82-87d5-b43ae1891805 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:30.355087927 +0000 UTC m=+1206.890995104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls") pod "feast-simple-feast-setup-57576b8fcf-r5wgm" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 11:50:29.855262 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:50:29.855121 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls podName:6d3b2737-dada-4b82-87d5-b43ae1891805 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:30.355112859 +0000 UTC m=+1206.891019834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-r5wgm" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805") : secret "feast-simple-feast-setup-registry-tls" not found Sep 11 11:50:29.855262 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:50:29.855136 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls podName:6d3b2737-dada-4b82-87d5-b43ae1891805 nodeName:}" failed. No retries permitted until 2026-09-11 11:50:30.355127485 +0000 UTC m=+1206.891034465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-r5wgm" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805") : secret "feast-simple-feast-setup-ui-tls" not found Sep 11 11:50:29.855262 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.855172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3b2737-dada-4b82-87d5-b43ae1891805-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.857386 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.857366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:29.863994 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:29.863971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89hp\" (UniqueName: \"kubernetes.io/projected/6d3b2737-dada-4b82-87d5-b43ae1891805-kube-api-access-p89hp\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.009804 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.009777 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerID="c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476" exitCode=0 Sep 11 11:50:30.009936 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.009848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerDied","Data":"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476"} Sep 11 11:50:30.359677 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.359638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.360019 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.359685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.360019 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.359708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.362117 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.362094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.362249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.362129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.362249 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.362158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-r5wgm\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.611389 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.611307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:30.885188 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:30.885162 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm"] Sep 11 11:50:30.886667 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:50:30.886640 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d3b2737_dada_4b82_87d5_b43ae1891805.slice/crio-08195e7f136e9799285839a07e1e82097b98529cc51a28cfe885bf2b526f1401 WatchSource:0}: Error finding container 08195e7f136e9799285839a07e1e82097b98529cc51a28cfe885bf2b526f1401: Status 404 returned error can't find the container with id 08195e7f136e9799285839a07e1e82097b98529cc51a28cfe885bf2b526f1401 Sep 11 11:50:31.017508 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:31.017473 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerID="7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad" exitCode=0 Sep 11 11:50:31.017640 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:31.017544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerDied","Data":"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad"} Sep 11 11:50:31.019124 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:31.019099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerStarted","Data":"42c49d0fe7815c533dd61c6922da340e4c7c9dbbc3792783539bc2d31b18f00e"} Sep 11 11:50:31.019236 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:31.019144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerStarted","Data":"08195e7f136e9799285839a07e1e82097b98529cc51a28cfe885bf2b526f1401"} Sep 11 11:50:36.041283 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:36.041209 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerID="42c49d0fe7815c533dd61c6922da340e4c7c9dbbc3792783539bc2d31b18f00e" exitCode=0 Sep 11 11:50:36.041607 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:36.041280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerDied","Data":"42c49d0fe7815c533dd61c6922da340e4c7c9dbbc3792783539bc2d31b18f00e"} Sep 11 11:50:37.047285 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:37.047250 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerStarted","Data":"dfad5fafc665f9289098fce2874c3640f2ac3618a4b6f7c047ec22a066e73244"} Sep 11 11:50:37.238609 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:37.238568 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 11 11:50:37.772805 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:37.772758 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 11 11:50:42.068505 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:42.068433 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerID="dfad5fafc665f9289098fce2874c3640f2ac3618a4b6f7c047ec22a066e73244" exitCode=0 Sep 11 11:50:42.069003 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:42.068510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerDied","Data":"dfad5fafc665f9289098fce2874c3640f2ac3618a4b6f7c047ec22a066e73244"} Sep 11 11:50:43.089180 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:43.089141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerStarted","Data":"c0a4e44b869e7a594a6804df97bee445b6c4a75620beba132322e2b2274cd864"} Sep 11 11:50:43.089699 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:43.089205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerStarted","Data":"e6450106bd25fdd1904b5bca95a9fd32f1309c2b762c305a109722639f4c21aa"} Sep 11 11:50:43.089699 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:43.089228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerStarted","Data":"2888498556ca187aca65bf4732c3dd15277115af1a571d9ab55d0d4c5d16b70e"} Sep 11 11:50:43.089699 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:43.089245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerStarted","Data":"b464af2b0e05bf40363e243037b5aa1856a5e740863bf0a4238680f057b4f668"} Sep 11 11:50:43.123873 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:43.123805 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podStartSLOduration=14.123782776 podStartE2EDuration="14.123782776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:43.122757001 +0000 UTC m=+1219.658663999" watchObservedRunningTime="2026-09-11 11:50:43.123782776 +0000 UTC m=+1219.659689775" Sep 11 11:50:45.612406 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.612365 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:45.613056 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.613026 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:45.613056 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.613062 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:45.613256 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.613080 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:45.614878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.614762 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Sep 11 11:50:45.614878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.614786 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 11 11:50:45.614878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.614805 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 11 11:50:45.614878 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:45.614841 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused" Sep 11 11:50:47.239002 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:47.238951 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 11 11:50:47.772807 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:47.772745 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 11 11:50:48.612017 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:48.611960 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 11 11:50:48.612479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:48.611960 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.35:8816: connect: connection refused" Sep 11 11:50:48.612479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:48.611974 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 11 11:50:48.612479 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:48.612211 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Sep 11 11:50:50.611814 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:50.611781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:50.612191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:50.611865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:50.612191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:50.611902 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:50.612191 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:50.611921 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:51.613013 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:51.612983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:51.613377 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:51.613112 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:51.613377 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:51.613266 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:51.618035 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:51.618015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:52.134324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:52.134296 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:52.134526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:52.134374 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:52.134526 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:52.134500 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:52.137024 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:52.137004 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:50:57.238752 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:57.238687 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.34:8443: connect: connection refused" Sep 11 11:50:57.239321 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:57.239010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:50:57.772149 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:57.772099 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" Sep 11 11:50:57.772352 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:57.772244 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:50:59.911517 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:50:59.911495 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:51:00.012611 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.012580 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-registry-tls\") pod \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " Sep 11 11:51:00.012611 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.012616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-offline-tls\") pod \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " Sep 11 11:51:00.012869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.012666 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mwdz\" (UniqueName: \"kubernetes.io/projected/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-kube-api-access-4mwdz\") pod \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " Sep 11 11:51:00.012869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.012705 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-online-tls\") pod \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " Sep 11 11:51:00.012869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.012756 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-ui-tls\") pod \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " Sep 11 11:51:00.012869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.012802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-feast-data\") pod \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\" (UID: \"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12\") " Sep 11 11:51:00.013732 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.013676 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-feast-data" (OuterVolumeSpecName: "feast-data") pod "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" (UID: "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:51:00.015179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.015143 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-online-tls" (OuterVolumeSpecName: "online-tls") pod "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" (UID: "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:51:00.015179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.015162 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" (UID: "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:51:00.015179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.015166 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-kube-api-access-4mwdz" (OuterVolumeSpecName: "kube-api-access-4mwdz") pod "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" (UID: "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12"). InnerVolumeSpecName "kube-api-access-4mwdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:51:00.015429 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.015414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" (UID: "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:51:00.015471 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.015433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" (UID: "5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:51:00.113850 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.113825 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.113850 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.113848 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.113988 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.113858 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.113988 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.113865 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.113988 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.113873 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwdz\" (UniqueName: \"kubernetes.io/projected/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-kube-api-access-4mwdz\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.113988 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.113880 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.165657 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.165629 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerID="45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a" exitCode=137 Sep 11 11:51:00.165657 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.165652 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerID="64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53" exitCode=137 Sep 11 11:51:00.165822 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.165744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerDied","Data":"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a"} Sep 11 11:51:00.165822 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.165763 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" Sep 11 11:51:00.165822 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.165781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerDied","Data":"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53"} Sep 11 11:51:00.165822 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.165795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg" event={"ID":"5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12","Type":"ContainerDied","Data":"fef48d4a01f3d1caba62b11c4e9473d211183395bf1c17b49e7f6b2b5619ff77"} Sep 11 11:51:00.165822 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.165824 2579 scope.go:117] "RemoveContainer" containerID="c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476" Sep 11 11:51:00.174245 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.174224 2579 scope.go:117] "RemoveContainer" containerID="45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a" Sep 11 11:51:00.183461 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.183445 2579 scope.go:117] "RemoveContainer" containerID="7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad" Sep 11 11:51:00.188761 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.188738 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg"] Sep 11 11:51:00.191970 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.191956 2579 scope.go:117] "RemoveContainer" containerID="64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53" Sep 11 11:51:00.193022 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.193003 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-fj9lg"] Sep 11 11:51:00.199287 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.199271 2579 scope.go:117] "RemoveContainer" containerID="506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228" Sep 11 11:51:00.206402 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.206388 2579 scope.go:117] "RemoveContainer" containerID="7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634" Sep 11 11:51:00.213240 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.213226 2579 scope.go:117] "RemoveContainer" containerID="c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476" Sep 11 11:51:00.213464 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:51:00.213449 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476\": container with ID starting with c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476 not found: ID does not exist" containerID="c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476" Sep 11 11:51:00.213515 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.213472 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476"} err="failed to get container status \"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476\": rpc error: code = NotFound desc = could not find container \"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476\": container with ID starting with c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476 not found: ID does not exist" Sep 11 11:51:00.213515 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.213493 2579 scope.go:117] "RemoveContainer" containerID="45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a" Sep 11 11:51:00.213732 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:51:00.213701 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a\": container with ID starting with 45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a not found: ID does not exist" containerID="45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a" Sep 11 11:51:00.213803 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.213744 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a"} err="failed to get container status \"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a\": rpc error: code = NotFound desc = could not find container \"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a\": container with ID starting with 45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a not found: ID does not exist" Sep 11 11:51:00.213803 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.213767 2579 scope.go:117] "RemoveContainer" containerID="7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad" Sep 11 11:51:00.214047 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:51:00.214031 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad\": container with ID starting with 7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad not found: ID does not exist" containerID="7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad" Sep 11 11:51:00.214109 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214055 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad"} err="failed to get container status \"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad\": rpc error: code = NotFound desc = could not find container \"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad\": container with ID starting with 7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad not found: ID does not exist" Sep 11 11:51:00.214109 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214075 2579 scope.go:117] "RemoveContainer" containerID="64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53" Sep 11 11:51:00.214355 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:51:00.214337 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53\": container with ID starting with 64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53 not found: ID does not exist" containerID="64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53" Sep 11 11:51:00.214399 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214362 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53"} err="failed to get container status \"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53\": rpc error: code = NotFound desc = could not find container \"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53\": container with ID starting with 64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53 not found: ID does not exist" Sep 11 11:51:00.214399 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214377 2579 scope.go:117] "RemoveContainer" containerID="506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228" Sep 11 11:51:00.214603 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:51:00.214587 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228\": container with ID starting with 506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228 not found: ID does not exist" containerID="506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228" Sep 11 11:51:00.214648 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214609 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228"} err="failed to get container status \"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228\": rpc error: code = NotFound desc = could not find container \"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228\": container with ID starting with 506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228 not found: ID does not exist" Sep 11 11:51:00.214648 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214623 2579 scope.go:117] "RemoveContainer" containerID="7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634" Sep 11 11:51:00.214912 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:51:00.214894 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634\": container with ID starting with 7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634 not found: ID does not exist" containerID="7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634" Sep 11 11:51:00.214960 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214915 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634"} err="failed to get container status \"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634\": rpc error: code = NotFound desc = could not find container \"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634\": container with ID starting with 7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634 not found: ID does not exist" Sep 11 11:51:00.214960 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.214930 2579 scope.go:117] "RemoveContainer" containerID="c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476" Sep 11 11:51:00.215149 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215133 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476"} err="failed to get container status \"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476\": rpc error: code = NotFound desc = could not find container \"c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476\": container with ID starting with c0265eaae1108ce869e7ac3738552d603d28d8c56579dd19162489de456de476 not found: ID does not exist" Sep 11 11:51:00.215247 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215149 2579 scope.go:117] "RemoveContainer" containerID="45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a" Sep 11 11:51:00.215414 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215399 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a"} err="failed to get container status \"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a\": rpc error: code = NotFound desc = could not find container \"45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a\": container with ID starting with 45b67dacd0b90b0fe4a6bc400d1b8c67c0110bfdf3b6338a7f422811053e245a not found: ID does not exist" Sep 11 11:51:00.215414 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215414 2579 scope.go:117] "RemoveContainer" containerID="7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad" Sep 11 11:51:00.215635 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215618 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad"} err="failed to get container status \"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad\": rpc error: code = NotFound desc = could not find container \"7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad\": container with ID starting with 7daee44bd97ad0aadbf5d0ca69394b8641d0f6a3131a0ac230d917e30630e8ad not found: ID does not exist" Sep 11 11:51:00.215635 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215635 2579 scope.go:117] "RemoveContainer" containerID="64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53" Sep 11 11:51:00.215917 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215902 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53"} err="failed to get container status \"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53\": rpc error: code = NotFound desc = could not find container \"64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53\": container with ID starting with 64e4f2e9f238b457ac6ffb16f40e110d910ef28ebc65ecca5156372385e22b53 not found: ID does not exist" Sep 11 11:51:00.215964 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.215917 2579 scope.go:117] "RemoveContainer" containerID="506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228" Sep 11 11:51:00.216124 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.216107 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228"} err="failed to get container status \"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228\": rpc error: code = NotFound desc = could not find container \"506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228\": container with ID starting with 506715f321e6a6fff3ab638774f4c1a4c46cb7b8a1b16c921fd90fe3a6af9228 not found: ID does not exist" Sep 11 11:51:00.216166 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.216125 2579 scope.go:117] "RemoveContainer" containerID="7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634" Sep 11 11:51:00.216332 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:00.216316 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634"} err="failed to get container status \"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634\": rpc error: code = NotFound desc = could not find container \"7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634\": container with ID starting with 7d86446a2c623acf20d90f9518bc957dc49196ced11835fca0e68f5aeabe7634 not found: ID does not exist" Sep 11 11:51:01.899366 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:01.899329 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" path="/var/lib/kubelet/pods/5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12/volumes" Sep 11 11:51:54.715664 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715628 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5"] Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715928 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="registry" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715940 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="registry" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715957 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="feast-apply" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715962 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="feast-apply" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715972 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715977 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715983 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715988 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.715996 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="feast-init" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.716001 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="feast-init" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.716009 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="offline" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.716014 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="offline" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.716064 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="offline" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.716074 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="registry" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.716084 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="ui" Sep 11 11:51:54.716148 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.716094 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1a0aa5-d4fb-4ce9-86d2-8522a98b6d12" containerName="online" Sep 11 11:51:54.719251 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.719233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.722985 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.722968 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 11:51:54.724244 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.724088 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 11:51:54.724244 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.724137 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 11:51:54.724244 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.724159 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 11:51:54.724244 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.724206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 11:51:54.724244 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.724161 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-wvvmn\"" Sep 11 11:51:54.724244 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.724140 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 11:51:54.732705 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.732684 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5"] Sep 11 11:51:54.796362 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.796340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93e971-0c3b-4dab-8d46-313490b26392-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.796464 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.796368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.796464 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.796385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.796588 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.796467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/ba93e971-0c3b-4dab-8d46-313490b26392-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.796588 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.796505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.796588 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.796541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgl4\" (UniqueName: \"kubernetes.io/projected/ba93e971-0c3b-4dab-8d46-313490b26392-kube-api-access-hbgl4\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897039 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/ba93e971-0c3b-4dab-8d46-313490b26392-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897151 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897151 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgl4\" (UniqueName: \"kubernetes.io/projected/ba93e971-0c3b-4dab-8d46-313490b26392-kube-api-access-hbgl4\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897151 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93e971-0c3b-4dab-8d46-313490b26392-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897315 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897579 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93e971-0c3b-4dab-8d46-313490b26392-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.897767 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.897747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/ba93e971-0c3b-4dab-8d46-313490b26392-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.899676 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.899655 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.899879 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.899859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.899924 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.899902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:54.905732 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:54.905694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgl4\" (UniqueName: \"kubernetes.io/projected/ba93e971-0c3b-4dab-8d46-313490b26392-kube-api-access-hbgl4\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-h67f5\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:55.027996 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:55.027925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:51:55.303887 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:55.303810 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5"] Sep 11 11:51:55.306792 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:51:55.306766 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba93e971_0c3b_4dab_8d46_313490b26392.slice/crio-04e315ac3c2b54992208f589a08aea6bd3ce1c0ac3932fffc22059338ff45265 WatchSource:0}: Error finding container 04e315ac3c2b54992208f589a08aea6bd3ce1c0ac3932fffc22059338ff45265: Status 404 returned error can't find the container with id 04e315ac3c2b54992208f589a08aea6bd3ce1c0ac3932fffc22059338ff45265 Sep 11 11:51:55.369651 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:55.369620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerStarted","Data":"04e315ac3c2b54992208f589a08aea6bd3ce1c0ac3932fffc22059338ff45265"} Sep 11 11:51:56.375501 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:51:56.375465 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerStarted","Data":"6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb"} Sep 11 11:52:00.392333 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:00.392300 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba93e971-0c3b-4dab-8d46-313490b26392" containerID="6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb" exitCode=0 Sep 11 11:52:00.392756 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:00.392375 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerDied","Data":"6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb"} Sep 11 11:52:01.398770 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:01.398738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerStarted","Data":"989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84"} Sep 11 11:52:29.508973 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:29.508939 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba93e971-0c3b-4dab-8d46-313490b26392" containerID="989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84" exitCode=0 Sep 11 11:52:29.509377 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:29.509015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerDied","Data":"989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84"} Sep 11 11:52:30.518300 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:30.518258 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerStarted","Data":"fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e"} Sep 11 11:52:30.518688 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:30.518327 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerStarted","Data":"fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb"} Sep 11 11:52:30.518688 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:30.518353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerStarted","Data":"1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0"} Sep 11 11:52:30.557110 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:30.557055 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podStartSLOduration=36.557039243 podStartE2EDuration="36.557039243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:52:30.554565406 +0000 UTC m=+1327.090472418" watchObservedRunningTime="2026-09-11 11:52:30.557039243 +0000 UTC m=+1327.092946241" Sep 11 11:52:31.028846 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:31.028806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:31.029033 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:31.028873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:31.029033 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:31.028893 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:31.030578 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:31.030534 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 11 11:52:31.030578 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:31.030555 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Sep 11 11:52:31.030893 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:31.030572 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 11 11:52:34.028967 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:34.028908 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 11 11:52:34.029458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:34.028920 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 11 11:52:34.029458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:34.028932 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Sep 11 11:52:35.029036 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:35.028996 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:35.029418 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:35.029059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:35.029418 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:35.029080 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:37.028598 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:37.028557 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.36:8816: connect: connection refused" Sep 11 11:52:37.028598 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:37.028593 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 11 11:52:37.029167 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:37.028561 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 11 11:52:40.029954 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:40.029925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:40.030409 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:40.030041 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:40.030699 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:40.030682 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:40.030833 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:40.030813 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:40.939911 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:40.939881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:52:42.438990 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:52:42.438957 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:53:45.220562 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.220528 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5"] Sep 11 11:53:45.221112 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.220821 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" containerID="cri-o://1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0" gracePeriod=30 Sep 11 11:53:45.221112 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.220869 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="offline" containerID="cri-o://fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb" gracePeriod=30 Sep 11 11:53:45.221112 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.220873 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" containerID="cri-o://fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e" gracePeriod=30 Sep 11 11:53:45.362949 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.362915 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm"] Sep 11 11:53:45.363215 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.363186 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="registry" containerID="cri-o://b464af2b0e05bf40363e243037b5aa1856a5e740863bf0a4238680f057b4f668" gracePeriod=30 Sep 11 11:53:45.363372 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.363296 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="offline" containerID="cri-o://e6450106bd25fdd1904b5bca95a9fd32f1309c2b762c305a109722639f4c21aa" gracePeriod=30 Sep 11 11:53:45.363372 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.363294 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" containerID="cri-o://c0a4e44b869e7a594a6804df97bee445b6c4a75620beba132322e2b2274cd864" gracePeriod=30 Sep 11 11:53:45.363372 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.363334 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" containerID="cri-o://2888498556ca187aca65bf4732c3dd15277115af1a571d9ab55d0d4c5d16b70e" gracePeriod=30 Sep 11 11:53:45.841302 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.841238 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba93e971-0c3b-4dab-8d46-313490b26392" containerID="fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e" exitCode=0 Sep 11 11:53:45.841633 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:45.841298 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerDied","Data":"fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e"} Sep 11 11:53:46.848231 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:46.848199 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba93e971-0c3b-4dab-8d46-313490b26392" containerID="1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0" exitCode=0 Sep 11 11:53:46.848634 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:46.848269 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerDied","Data":"1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0"} Sep 11 11:53:46.851179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:46.851156 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerID="c0a4e44b869e7a594a6804df97bee445b6c4a75620beba132322e2b2274cd864" exitCode=0 Sep 11 11:53:46.851179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:46.851176 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerID="2888498556ca187aca65bf4732c3dd15277115af1a571d9ab55d0d4c5d16b70e" exitCode=0 Sep 11 11:53:46.851346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:46.851215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerDied","Data":"c0a4e44b869e7a594a6804df97bee445b6c4a75620beba132322e2b2274cd864"} Sep 11 11:53:46.851346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:46.851257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerDied","Data":"2888498556ca187aca65bf4732c3dd15277115af1a571d9ab55d0d4c5d16b70e"} Sep 11 11:53:50.031184 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:50.031133 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 11 11:53:51.580412 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:51.580369 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 11 11:53:52.134246 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:52.134202 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 11 11:53:52.134422 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:53:52.134264 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 11 11:54:00.030755 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:00.030681 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 11 11:54:01.581006 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:01.580958 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 11 11:54:02.134799 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:02.134756 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 11 11:54:02.134985 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:02.134775 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 11 11:54:10.030801 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:10.030754 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.36:8443: connect: connection refused" Sep 11 11:54:10.031236 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:10.030918 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:54:11.581098 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:11.581051 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 11 11:54:11.581506 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:11.581194 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:54:12.133977 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:12.133932 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.35:8443: connect: connection refused" Sep 11 11:54:12.134179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:12.133935 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 11 11:54:12.134179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:12.134079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:54:12.134308 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:12.134276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:54:15.861845 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.861823 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:54:15.896764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.896731 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/ba93e971-0c3b-4dab-8d46-313490b26392-registry-tls\") pod \"ba93e971-0c3b-4dab-8d46-313490b26392\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " Sep 11 11:54:15.896981 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.896790 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-offline-tls\") pod \"ba93e971-0c3b-4dab-8d46-313490b26392\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " Sep 11 11:54:15.896981 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.896834 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-ui-tls\") pod \"ba93e971-0c3b-4dab-8d46-313490b26392\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " Sep 11 11:54:15.896981 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.896882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93e971-0c3b-4dab-8d46-313490b26392-feast-data\") pod \"ba93e971-0c3b-4dab-8d46-313490b26392\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " Sep 11 11:54:15.896981 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.896912 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-online-tls\") pod \"ba93e971-0c3b-4dab-8d46-313490b26392\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " Sep 11 11:54:15.896981 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.896944 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbgl4\" (UniqueName: \"kubernetes.io/projected/ba93e971-0c3b-4dab-8d46-313490b26392-kube-api-access-hbgl4\") pod \"ba93e971-0c3b-4dab-8d46-313490b26392\" (UID: \"ba93e971-0c3b-4dab-8d46-313490b26392\") " Sep 11 11:54:15.897512 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.897406 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba93e971-0c3b-4dab-8d46-313490b26392-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ba93e971-0c3b-4dab-8d46-313490b26392" (UID: "ba93e971-0c3b-4dab-8d46-313490b26392"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:54:15.897632 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.897541 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba93e971-0c3b-4dab-8d46-313490b26392-feast-data" (OuterVolumeSpecName: "feast-data") pod "ba93e971-0c3b-4dab-8d46-313490b26392" (UID: "ba93e971-0c3b-4dab-8d46-313490b26392"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:54:15.900033 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.899985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba93e971-0c3b-4dab-8d46-313490b26392-kube-api-access-hbgl4" (OuterVolumeSpecName: "kube-api-access-hbgl4") pod "ba93e971-0c3b-4dab-8d46-313490b26392" (UID: "ba93e971-0c3b-4dab-8d46-313490b26392"). InnerVolumeSpecName "kube-api-access-hbgl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:54:15.900140 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.900089 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ba93e971-0c3b-4dab-8d46-313490b26392" (UID: "ba93e971-0c3b-4dab-8d46-313490b26392"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:54:15.900140 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.900105 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-online-tls" (OuterVolumeSpecName: "online-tls") pod "ba93e971-0c3b-4dab-8d46-313490b26392" (UID: "ba93e971-0c3b-4dab-8d46-313490b26392"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:54:15.900140 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.900123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ba93e971-0c3b-4dab-8d46-313490b26392" (UID: "ba93e971-0c3b-4dab-8d46-313490b26392"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:54:15.962884 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.962857 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba93e971-0c3b-4dab-8d46-313490b26392" containerID="fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb" exitCode=137 Sep 11 11:54:15.963017 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.962920 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" Sep 11 11:54:15.963017 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.962944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerDied","Data":"fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb"} Sep 11 11:54:15.963017 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.962993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5" event={"ID":"ba93e971-0c3b-4dab-8d46-313490b26392","Type":"ContainerDied","Data":"04e315ac3c2b54992208f589a08aea6bd3ce1c0ac3932fffc22059338ff45265"} Sep 11 11:54:15.963161 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.963019 2579 scope.go:117] "RemoveContainer" containerID="fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e" Sep 11 11:54:15.966834 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.966784 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerID="e6450106bd25fdd1904b5bca95a9fd32f1309c2b762c305a109722639f4c21aa" exitCode=137 Sep 11 11:54:15.966834 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.966803 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerID="b464af2b0e05bf40363e243037b5aa1856a5e740863bf0a4238680f057b4f668" exitCode=137 Sep 11 11:54:15.967067 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.966838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerDied","Data":"e6450106bd25fdd1904b5bca95a9fd32f1309c2b762c305a109722639f4c21aa"} Sep 11 11:54:15.967067 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.966868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerDied","Data":"b464af2b0e05bf40363e243037b5aa1856a5e740863bf0a4238680f057b4f668"} Sep 11 11:54:15.971862 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.971843 2579 scope.go:117] "RemoveContainer" containerID="fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb" Sep 11 11:54:15.979036 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.979017 2579 scope.go:117] "RemoveContainer" containerID="1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0" Sep 11 11:54:15.984093 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.984073 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5"] Sep 11 11:54:15.987018 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.987001 2579 scope.go:117] "RemoveContainer" containerID="989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84" Sep 11 11:54:15.990324 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.990301 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-h67f5"] Sep 11 11:54:15.995470 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.995437 2579 scope.go:117] "RemoveContainer" containerID="6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb" Sep 11 11:54:15.997685 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.997668 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/ba93e971-0c3b-4dab-8d46-313490b26392-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:15.997685 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.997687 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:15.997888 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.997696 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:15.997888 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.997704 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba93e971-0c3b-4dab-8d46-313490b26392-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:15.997888 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.997711 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba93e971-0c3b-4dab-8d46-313490b26392-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:15.997888 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:15.997811 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbgl4\" (UniqueName: \"kubernetes.io/projected/ba93e971-0c3b-4dab-8d46-313490b26392-kube-api-access-hbgl4\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:16.004643 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.004615 2579 scope.go:117] "RemoveContainer" containerID="fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e" Sep 11 11:54:16.005003 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:16.004972 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e\": container with ID starting with fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e not found: ID does not exist" containerID="fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e" Sep 11 11:54:16.005099 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005013 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e"} err="failed to get container status \"fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e\": rpc error: code = NotFound desc = could not find container \"fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e\": container with ID starting with fec22a5d388a556c956550f3d84450c33f2c955064bc47877b71393fe0f2e97e not found: ID does not exist" Sep 11 11:54:16.005099 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005041 2579 scope.go:117] "RemoveContainer" containerID="fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb" Sep 11 11:54:16.005346 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:16.005325 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb\": container with ID starting with fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb not found: ID does not exist" containerID="fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb" Sep 11 11:54:16.005396 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005355 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb"} err="failed to get container status \"fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb\": rpc error: code = NotFound desc = could not find container \"fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb\": container with ID starting with fe66fe3ccb58b9e32db351b73726b8cfd638cf610a861f6edb85426f0d7400eb not found: ID does not exist" Sep 11 11:54:16.005396 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005375 2579 scope.go:117] "RemoveContainer" containerID="1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0" Sep 11 11:54:16.005652 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:16.005635 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0\": container with ID starting with 1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0 not found: ID does not exist" containerID="1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0" Sep 11 11:54:16.005698 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005664 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0"} err="failed to get container status \"1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0\": rpc error: code = NotFound desc = could not find container \"1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0\": container with ID starting with 1f5c65474d9fd82c045788cea9711493494c3115e3fc343d7c2f3b2c16810df0 not found: ID does not exist" Sep 11 11:54:16.005698 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005678 2579 scope.go:117] "RemoveContainer" containerID="989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84" Sep 11 11:54:16.005942 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:16.005925 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84\": container with ID starting with 989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84 not found: ID does not exist" containerID="989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84" Sep 11 11:54:16.006007 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005954 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84"} err="failed to get container status \"989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84\": rpc error: code = NotFound desc = could not find container \"989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84\": container with ID starting with 989a53957b087f17e8b1c8f2162e0878fb612515e47a976cbfb8f9c083169f84 not found: ID does not exist" Sep 11 11:54:16.006007 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.005980 2579 scope.go:117] "RemoveContainer" containerID="6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb" Sep 11 11:54:16.006278 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:16.006263 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb\": container with ID starting with 6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb not found: ID does not exist" containerID="6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb" Sep 11 11:54:16.006334 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.006283 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb"} err="failed to get container status \"6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb\": rpc error: code = NotFound desc = could not find container \"6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb\": container with ID starting with 6628bf4dfb603981788742351b239f123d05dd7f19bfd2031aa7ca941ae7a9fb not found: ID does not exist" Sep 11 11:54:16.013154 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.013136 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:54:16.098214 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.098191 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3b2737-dada-4b82-87d5-b43ae1891805-feast-data\") pod \"6d3b2737-dada-4b82-87d5-b43ae1891805\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " Sep 11 11:54:16.098346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.098226 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p89hp\" (UniqueName: \"kubernetes.io/projected/6d3b2737-dada-4b82-87d5-b43ae1891805-kube-api-access-p89hp\") pod \"6d3b2737-dada-4b82-87d5-b43ae1891805\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " Sep 11 11:54:16.098346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.098276 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls\") pod \"6d3b2737-dada-4b82-87d5-b43ae1891805\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " Sep 11 11:54:16.098346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.098296 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls\") pod \"6d3b2737-dada-4b82-87d5-b43ae1891805\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " Sep 11 11:54:16.098346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.098320 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls\") pod \"6d3b2737-dada-4b82-87d5-b43ae1891805\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " Sep 11 11:54:16.098346 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.098342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-offline-tls\") pod \"6d3b2737-dada-4b82-87d5-b43ae1891805\" (UID: \"6d3b2737-dada-4b82-87d5-b43ae1891805\") " Sep 11 11:54:16.098772 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.098743 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3b2737-dada-4b82-87d5-b43ae1891805-feast-data" (OuterVolumeSpecName: "feast-data") pod "6d3b2737-dada-4b82-87d5-b43ae1891805" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:54:16.100665 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.100642 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6d3b2737-dada-4b82-87d5-b43ae1891805" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:54:16.100774 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.100671 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls" (OuterVolumeSpecName: "online-tls") pod "6d3b2737-dada-4b82-87d5-b43ae1891805" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:54:16.100774 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.100679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6d3b2737-dada-4b82-87d5-b43ae1891805" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:54:16.100972 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.100952 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "6d3b2737-dada-4b82-87d5-b43ae1891805" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:54:16.100972 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.100964 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3b2737-dada-4b82-87d5-b43ae1891805-kube-api-access-p89hp" (OuterVolumeSpecName: "kube-api-access-p89hp") pod "6d3b2737-dada-4b82-87d5-b43ae1891805" (UID: "6d3b2737-dada-4b82-87d5-b43ae1891805"). InnerVolumeSpecName "kube-api-access-p89hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:54:16.198869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.198847 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:16.198869 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.198869 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:16.199018 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.198877 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3b2737-dada-4b82-87d5-b43ae1891805-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:16.199018 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.198885 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p89hp\" (UniqueName: \"kubernetes.io/projected/6d3b2737-dada-4b82-87d5-b43ae1891805-kube-api-access-p89hp\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:16.199018 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.198893 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:16.199018 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.198900 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/6d3b2737-dada-4b82-87d5-b43ae1891805-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:54:16.973338 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.973310 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" Sep 11 11:54:16.973338 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.973320 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm" event={"ID":"6d3b2737-dada-4b82-87d5-b43ae1891805","Type":"ContainerDied","Data":"08195e7f136e9799285839a07e1e82097b98529cc51a28cfe885bf2b526f1401"} Sep 11 11:54:16.973848 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.973380 2579 scope.go:117] "RemoveContainer" containerID="c0a4e44b869e7a594a6804df97bee445b6c4a75620beba132322e2b2274cd864" Sep 11 11:54:16.981977 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.981960 2579 scope.go:117] "RemoveContainer" containerID="e6450106bd25fdd1904b5bca95a9fd32f1309c2b762c305a109722639f4c21aa" Sep 11 11:54:16.989610 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.989594 2579 scope.go:117] "RemoveContainer" containerID="2888498556ca187aca65bf4732c3dd15277115af1a571d9ab55d0d4c5d16b70e" Sep 11 11:54:16.996602 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.996582 2579 scope.go:117] "RemoveContainer" containerID="b464af2b0e05bf40363e243037b5aa1856a5e740863bf0a4238680f057b4f668" Sep 11 11:54:16.999520 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:16.999497 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm"] Sep 11 11:54:17.002486 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:17.002466 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-r5wgm"] Sep 11 11:54:17.004866 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:17.004852 2579 scope.go:117] "RemoveContainer" containerID="dfad5fafc665f9289098fce2874c3640f2ac3618a4b6f7c047ec22a066e73244" Sep 11 11:54:17.011764 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:17.011745 2579 scope.go:117] "RemoveContainer" containerID="42c49d0fe7815c533dd61c6922da340e4c7c9dbbc3792783539bc2d31b18f00e" Sep 11 11:54:17.898781 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:17.898741 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" path="/var/lib/kubelet/pods/6d3b2737-dada-4b82-87d5-b43ae1891805/volumes" Sep 11 11:54:17.899413 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:17.899391 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" path="/var/lib/kubelet/pods/ba93e971-0c3b-4dab-8d46-313490b26392/volumes" Sep 11 11:54:57.414377 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414340 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj"] Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414607 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="feast-apply" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414618 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="feast-apply" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414628 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414633 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414640 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414645 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414651 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414657 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414663 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414668 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414674 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="registry" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414679 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="registry" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414688 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="feast-init" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414694 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="feast-init" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414701 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="offline" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414706 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="offline" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414726 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="feast-init" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414731 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="feast-init" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414743 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="offline" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414752 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="offline" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414762 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="feast-apply" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414767 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="feast-apply" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414808 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="registry" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414814 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="offline" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414819 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="online" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414833 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="offline" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414841 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba93e971-0c3b-4dab-8d46-313490b26392" containerName="ui" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414846 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="ui" Sep 11 11:54:57.414881 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.414851 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3b2737-dada-4b82-87d5-b43ae1891805" containerName="online" Sep 11 11:54:57.418032 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.418016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.420864 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.420839 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 11 11:54:57.420864 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.420851 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 11 11:54:57.421127 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.420875 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 11 11:54:57.421127 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.420899 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 11 11:54:57.421127 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.420851 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 11 11:54:57.421127 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.420850 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-nxssn\"" Sep 11 11:54:57.421353 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.421256 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 11 11:54:57.430773 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.430751 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj"] Sep 11 11:54:57.504083 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.504059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlpq\" (UniqueName: \"kubernetes.io/projected/5f3b47b5-c32f-429f-b1ca-525f93117f37-kube-api-access-tjlpq\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.504230 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.504090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.504230 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.504111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.504230 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.504141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.504230 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.504170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.504230 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.504193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3b47b5-c32f-429f-b1ca-525f93117f37-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605501 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.605475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3b47b5-c32f-429f-b1ca-525f93117f37-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605625 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.605529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlpq\" (UniqueName: \"kubernetes.io/projected/5f3b47b5-c32f-429f-b1ca-525f93117f37-kube-api-access-tjlpq\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605665 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.605645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605705 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.605693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605772 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.605752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605811 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.605783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605811 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:57.605796 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 11 11:54:57.605885 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:57.605827 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 11 11:54:57.605885 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:57.605866 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 11 11:54:57.605966 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:57.605873 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls podName:5f3b47b5-c32f-429f-b1ca-525f93117f37 nodeName:}" failed. No retries permitted until 2026-09-11 11:54:58.105857491 +0000 UTC m=+1474.641764468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls") pod "feast-simple-feast-setup-57576b8fcf-l7jmj" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37") : secret "feast-simple-feast-setup-online-tls" not found Sep 11 11:54:57.605966 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.605928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3b47b5-c32f-429f-b1ca-525f93117f37-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.605966 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:57.605947 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls podName:5f3b47b5-c32f-429f-b1ca-525f93117f37 nodeName:}" failed. No retries permitted until 2026-09-11 11:54:58.105930769 +0000 UTC m=+1474.641837749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-l7jmj" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37") : secret "feast-simple-feast-setup-ui-tls" not found Sep 11 11:54:57.606088 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:54:57.605970 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls podName:5f3b47b5-c32f-429f-b1ca-525f93117f37 nodeName:}" failed. No retries permitted until 2026-09-11 11:54:58.105959351 +0000 UTC m=+1474.641866328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-l7jmj" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37") : secret "feast-simple-feast-setup-offline-tls" not found Sep 11 11:54:57.608253 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.608235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:57.614243 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:57.614223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlpq\" (UniqueName: \"kubernetes.io/projected/5f3b47b5-c32f-429f-b1ca-525f93117f37-kube-api-access-tjlpq\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.109448 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.109403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.109448 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.109455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.109751 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.109485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.112063 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.112029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.112184 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.112086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.112184 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.112099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-l7jmj\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.327008 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.326966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:54:58.601494 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:58.601470 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj"] Sep 11 11:54:58.602700 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:54:58.602675 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f3b47b5_c32f_429f_b1ca_525f93117f37.slice/crio-701cb3bb0361967d07e2829f6dc4d0fa90d4b9ad5defc1976a803fb239c4f668 WatchSource:0}: Error finding container 701cb3bb0361967d07e2829f6dc4d0fa90d4b9ad5defc1976a803fb239c4f668: Status 404 returned error can't find the container with id 701cb3bb0361967d07e2829f6dc4d0fa90d4b9ad5defc1976a803fb239c4f668 Sep 11 11:54:59.131326 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:59.131285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerStarted","Data":"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9"} Sep 11 11:54:59.131505 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:54:59.131342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerStarted","Data":"701cb3bb0361967d07e2829f6dc4d0fa90d4b9ad5defc1976a803fb239c4f668"} Sep 11 11:55:04.156840 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:04.156807 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerID="a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9" exitCode=0 Sep 11 11:55:04.157371 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:04.156880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerDied","Data":"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9"} Sep 11 11:55:05.162439 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:05.162402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerStarted","Data":"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0"} Sep 11 11:55:10.183071 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:10.183034 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerID="4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0" exitCode=0 Sep 11 11:55:10.183489 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:10.183101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerDied","Data":"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0"} Sep 11 11:55:11.193586 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:11.193536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerStarted","Data":"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530"} Sep 11 11:55:11.194103 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:11.193593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerStarted","Data":"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508"} Sep 11 11:55:11.194103 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:11.193616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerStarted","Data":"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce"} Sep 11 11:55:11.194103 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:11.193634 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerStarted","Data":"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac"} Sep 11 11:55:11.220475 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:11.220404 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podStartSLOduration=14.220380955 podStartE2EDuration="14.220380955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:55:11.216459619 +0000 UTC m=+1487.752366618" watchObservedRunningTime="2026-09-11 11:55:11.220380955 +0000 UTC m=+1487.756287955" Sep 11 11:55:13.327636 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.327576 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:13.328164 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.327649 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:13.328164 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.327672 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:13.328164 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.327691 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:13.329649 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.329555 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Sep 11 11:55:13.329649 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.329571 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Sep 11 11:55:13.329649 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.329581 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 11 11:55:13.329952 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:13.329659 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 11 11:55:16.327629 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:16.327569 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 11 11:55:16.328308 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:16.327662 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 11 11:55:16.328308 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:16.327572 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Sep 11 11:55:16.328308 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:16.327582 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Sep 11 11:55:18.327699 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:18.327667 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:18.328075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:18.327739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:18.328075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:18.327753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:18.328075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:18.327764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:19.327962 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:19.327924 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:19.328394 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:19.328070 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:19.328394 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:19.328329 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:19.328887 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:19.328858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:19.329061 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:19.329043 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:19.329317 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:19.329303 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:19.333664 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:19.333644 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:55:20.241794 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:55:20.241764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:56:21.967537 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:21.967405 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj"] Sep 11 11:56:21.970240 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:21.967848 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="registry" containerID="cri-o://92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac" gracePeriod=30 Sep 11 11:56:21.970240 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:21.967927 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" containerID="cri-o://c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce" gracePeriod=30 Sep 11 11:56:21.970240 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:21.967924 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="offline" containerID="cri-o://f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508" gracePeriod=30 Sep 11 11:56:21.970240 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:21.967951 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" containerID="cri-o://499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530" gracePeriod=30 Sep 11 11:56:22.404982 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.404941 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4"] Sep 11 11:56:22.408492 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.408470 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.411549 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.411517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-lm5n7\"" Sep 11 11:56:22.421979 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.421952 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4"] Sep 11 11:56:22.551255 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.551218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnnr\" (UniqueName: \"kubernetes.io/projected/4d5e5f6d-af5b-497f-b952-0f1db866398d-kube-api-access-fpnnr\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.551455 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.551285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.551455 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.551309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.551455 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.551342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5e5f6d-af5b-497f-b952-0f1db866398d-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.551455 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.551367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.551656 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.551454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.652287 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.652258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.652456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.652296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.652456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.652320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5e5f6d-af5b-497f-b952-0f1db866398d-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.652456 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:22.652419 2579 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 11 11:56:22.652456 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.652441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.652677 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.652495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.652677 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.652573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnnr\" (UniqueName: \"kubernetes.io/projected/4d5e5f6d-af5b-497f-b952-0f1db866398d-kube-api-access-fpnnr\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.652832 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:22.652772 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls podName:4d5e5f6d-af5b-497f-b952-0f1db866398d nodeName:}" failed. No retries permitted until 2026-09-11 11:56:23.152748218 +0000 UTC m=+1559.688655208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-7wkm4" (UID: "4d5e5f6d-af5b-497f-b952-0f1db866398d") : secret "feast-simple-feast-setup-ui-tls" not found Sep 11 11:56:22.652907 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.652834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5e5f6d-af5b-497f-b952-0f1db866398d-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.654967 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.654937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.655091 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.655025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.655166 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.655146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:22.661759 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:22.661734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnnr\" (UniqueName: \"kubernetes.io/projected/4d5e5f6d-af5b-497f-b952-0f1db866398d-kube-api-access-fpnnr\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:23.157402 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.157375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:23.159957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.159933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-7wkm4\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:23.319704 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.319677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:23.476687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.476606 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerID="499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530" exitCode=0 Sep 11 11:56:23.476687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.476637 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerID="c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce" exitCode=0 Sep 11 11:56:23.476896 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.476679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerDied","Data":"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530"} Sep 11 11:56:23.476896 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.476762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerDied","Data":"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce"} Sep 11 11:56:23.593834 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:23.593808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4"] Sep 11 11:56:23.594504 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:56:23.594479 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5e5f6d_af5b_497f_b952_0f1db866398d.slice/crio-f7065e0f4b3033ea3be47b312888e67305e02602848674adb8ba29e4e06bbeae WatchSource:0}: Error finding container f7065e0f4b3033ea3be47b312888e67305e02602848674adb8ba29e4e06bbeae: Status 404 returned error can't find the container with id f7065e0f4b3033ea3be47b312888e67305e02602848674adb8ba29e4e06bbeae Sep 11 11:56:24.482584 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:24.482551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerStarted","Data":"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6"} Sep 11 11:56:24.483006 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:24.482593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerStarted","Data":"f7065e0f4b3033ea3be47b312888e67305e02602848674adb8ba29e4e06bbeae"} Sep 11 11:56:29.329031 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:29.328975 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 11 11:56:29.503177 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:29.503140 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerID="d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6" exitCode=0 Sep 11 11:56:29.503348 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:29.503211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerDied","Data":"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6"} Sep 11 11:56:30.239573 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:30.239524 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 11 11:56:30.509686 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:30.509598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerStarted","Data":"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7"} Sep 11 11:56:35.529915 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:35.529824 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerID="db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7" exitCode=0 Sep 11 11:56:35.529915 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:35.529893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerDied","Data":"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7"} Sep 11 11:56:36.540233 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:36.540189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerStarted","Data":"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6"} Sep 11 11:56:36.540753 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:36.540259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerStarted","Data":"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3"} Sep 11 11:56:36.540753 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:36.540280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerStarted","Data":"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab"} Sep 11 11:56:36.540753 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:36.540299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerStarted","Data":"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0"} Sep 11 11:56:36.568031 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:36.567971 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podStartSLOduration=14.567948493 podStartE2EDuration="14.567948493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:56:36.56652651 +0000 UTC m=+1573.102433510" watchObservedRunningTime="2026-09-11 11:56:36.567948493 +0000 UTC m=+1573.103855494" Sep 11 11:56:38.320631 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.320577 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:38.321159 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.320654 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:38.321159 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.320676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:38.321159 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.320694 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:38.322788 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.322751 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused" Sep 11 11:56:38.323028 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.322776 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 11 11:56:38.323116 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.322796 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.38:8816: connect: connection refused" Sep 11 11:56:38.323116 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:38.322799 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 11 11:56:39.328820 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:39.328774 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 11 11:56:40.239230 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:40.239178 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 11 11:56:41.320205 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:41.320161 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused" Sep 11 11:56:41.320813 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:41.320168 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 11 11:56:41.320964 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:41.320158 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.38:8816: connect: connection refused" Sep 11 11:56:41.320964 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:41.320921 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 11 11:56:43.320520 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:43.320482 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:43.321042 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:43.320552 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:43.321042 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:43.320565 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:43.321042 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:43.320585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.320992 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.320959 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.321434 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.321154 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.321434 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.321400 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.326297 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.326276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.583257 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.583182 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.583257 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.583248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.583434 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.583377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:44.585672 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:44.585653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:56:49.329281 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:49.329232 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 11 11:56:49.329766 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:49.329377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:56:50.239130 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:50.239096 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 11 11:56:50.239294 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:50.239228 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:56:52.611835 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.611807 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:56:52.612769 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.612741 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerID="f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508" exitCode=137 Sep 11 11:56:52.612883 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.612774 2579 generic.go:356] "Generic (PLEG): container finished" podID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerID="92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac" exitCode=137 Sep 11 11:56:52.612883 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.612746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerDied","Data":"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508"} Sep 11 11:56:52.612883 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.612829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerDied","Data":"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac"} Sep 11 11:56:52.612883 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.612843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" event={"ID":"5f3b47b5-c32f-429f-b1ca-525f93117f37","Type":"ContainerDied","Data":"701cb3bb0361967d07e2829f6dc4d0fa90d4b9ad5defc1976a803fb239c4f668"} Sep 11 11:56:52.612883 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.612861 2579 scope.go:117] "RemoveContainer" containerID="499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530" Sep 11 11:56:52.621214 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.621195 2579 scope.go:117] "RemoveContainer" containerID="f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508" Sep 11 11:56:52.628352 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.628329 2579 scope.go:117] "RemoveContainer" containerID="c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce" Sep 11 11:56:52.635639 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.635618 2579 scope.go:117] "RemoveContainer" containerID="92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac" Sep 11 11:56:52.643591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.643483 2579 scope.go:117] "RemoveContainer" containerID="4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0" Sep 11 11:56:52.650858 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.650838 2579 scope.go:117] "RemoveContainer" containerID="a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9" Sep 11 11:56:52.658060 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.658038 2579 scope.go:117] "RemoveContainer" containerID="499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530" Sep 11 11:56:52.658352 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:52.658324 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530\": container with ID starting with 499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530 not found: ID does not exist" containerID="499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530" Sep 11 11:56:52.658399 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.658363 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530"} err="failed to get container status \"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530\": rpc error: code = NotFound desc = could not find container \"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530\": container with ID starting with 499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530 not found: ID does not exist" Sep 11 11:56:52.658399 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.658382 2579 scope.go:117] "RemoveContainer" containerID="f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508" Sep 11 11:56:52.658665 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:52.658648 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508\": container with ID starting with f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508 not found: ID does not exist" containerID="f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508" Sep 11 11:56:52.658741 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.658678 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508"} err="failed to get container status \"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508\": rpc error: code = NotFound desc = could not find container \"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508\": container with ID starting with f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508 not found: ID does not exist" Sep 11 11:56:52.658741 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.658694 2579 scope.go:117] "RemoveContainer" containerID="c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce" Sep 11 11:56:52.658961 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:52.658943 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce\": container with ID starting with c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce not found: ID does not exist" containerID="c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce" Sep 11 11:56:52.659010 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.658966 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce"} err="failed to get container status \"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce\": rpc error: code = NotFound desc = could not find container \"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce\": container with ID starting with c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce not found: ID does not exist" Sep 11 11:56:52.659010 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.658980 2579 scope.go:117] "RemoveContainer" containerID="92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac" Sep 11 11:56:52.659250 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:52.659233 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac\": container with ID starting with 92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac not found: ID does not exist" containerID="92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac" Sep 11 11:56:52.659295 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.659255 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac"} err="failed to get container status \"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac\": rpc error: code = NotFound desc = could not find container \"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac\": container with ID starting with 92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac not found: ID does not exist" Sep 11 11:56:52.659295 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.659270 2579 scope.go:117] "RemoveContainer" containerID="4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0" Sep 11 11:56:52.659517 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:52.659496 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0\": container with ID starting with 4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0 not found: ID does not exist" containerID="4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0" Sep 11 11:56:52.659576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.659527 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0"} err="failed to get container status \"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0\": rpc error: code = NotFound desc = could not find container \"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0\": container with ID starting with 4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0 not found: ID does not exist" Sep 11 11:56:52.659576 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.659550 2579 scope.go:117] "RemoveContainer" containerID="a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9" Sep 11 11:56:52.659804 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:56:52.659786 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9\": container with ID starting with a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9 not found: ID does not exist" containerID="a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9" Sep 11 11:56:52.659872 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.659806 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9"} err="failed to get container status \"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9\": rpc error: code = NotFound desc = could not find container \"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9\": container with ID starting with a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9 not found: ID does not exist" Sep 11 11:56:52.659872 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.659824 2579 scope.go:117] "RemoveContainer" containerID="499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530" Sep 11 11:56:52.660063 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660033 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530"} err="failed to get container status \"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530\": rpc error: code = NotFound desc = could not find container \"499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530\": container with ID starting with 499c961ea29ddc0e44379dec78641f71aa3d754c22716111db1c7b5159403530 not found: ID does not exist" Sep 11 11:56:52.660105 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660064 2579 scope.go:117] "RemoveContainer" containerID="f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508" Sep 11 11:56:52.660311 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660292 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508"} err="failed to get container status \"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508\": rpc error: code = NotFound desc = could not find container \"f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508\": container with ID starting with f18433f96444a1e84b23698303ff8cd82f8a9f542a8af398a76778e955d13508 not found: ID does not exist" Sep 11 11:56:52.660354 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660312 2579 scope.go:117] "RemoveContainer" containerID="c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce" Sep 11 11:56:52.660574 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660557 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce"} err="failed to get container status \"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce\": rpc error: code = NotFound desc = could not find container \"c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce\": container with ID starting with c89fa465f1883dde25e35e9d5820357f608a91ce504e113311544e598e79a9ce not found: ID does not exist" Sep 11 11:56:52.660624 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660575 2579 scope.go:117] "RemoveContainer" containerID="92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac" Sep 11 11:56:52.660810 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660793 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac"} err="failed to get container status \"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac\": rpc error: code = NotFound desc = could not find container \"92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac\": container with ID starting with 92bd0b2a39ad964fee6959b0994a1217246492f50f4cab1ed82e66483d1425ac not found: ID does not exist" Sep 11 11:56:52.660858 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.660811 2579 scope.go:117] "RemoveContainer" containerID="4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0" Sep 11 11:56:52.661046 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.661029 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0"} err="failed to get container status \"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0\": rpc error: code = NotFound desc = could not find container \"4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0\": container with ID starting with 4d76d8a7cb53be8e267fcb1f8c15df9630f1029d5708b9c6d27da3204e14a0c0 not found: ID does not exist" Sep 11 11:56:52.661106 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.661047 2579 scope.go:117] "RemoveContainer" containerID="a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9" Sep 11 11:56:52.661278 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.661259 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9"} err="failed to get container status \"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9\": rpc error: code = NotFound desc = could not find container \"a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9\": container with ID starting with a45d04418314384035fa5c2d352bd4a8c163efb1d99e020a9140e351d86ce2a9 not found: ID does not exist" Sep 11 11:56:52.689565 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.689542 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjlpq\" (UniqueName: \"kubernetes.io/projected/5f3b47b5-c32f-429f-b1ca-525f93117f37-kube-api-access-tjlpq\") pod \"5f3b47b5-c32f-429f-b1ca-525f93117f37\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " Sep 11 11:56:52.689641 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.689569 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-registry-tls\") pod \"5f3b47b5-c32f-429f-b1ca-525f93117f37\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " Sep 11 11:56:52.689641 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.689604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls\") pod \"5f3b47b5-c32f-429f-b1ca-525f93117f37\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " Sep 11 11:56:52.689709 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.689660 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls\") pod \"5f3b47b5-c32f-429f-b1ca-525f93117f37\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " Sep 11 11:56:52.689709 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.689690 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3b47b5-c32f-429f-b1ca-525f93117f37-feast-data\") pod \"5f3b47b5-c32f-429f-b1ca-525f93117f37\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " Sep 11 11:56:52.689849 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.689707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls\") pod \"5f3b47b5-c32f-429f-b1ca-525f93117f37\" (UID: \"5f3b47b5-c32f-429f-b1ca-525f93117f37\") " Sep 11 11:56:52.690221 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.690198 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3b47b5-c32f-429f-b1ca-525f93117f37-feast-data" (OuterVolumeSpecName: "feast-data") pod "5f3b47b5-c32f-429f-b1ca-525f93117f37" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:56:52.692123 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.692089 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls" (OuterVolumeSpecName: "online-tls") pod "5f3b47b5-c32f-429f-b1ca-525f93117f37" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:56:52.692281 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.692265 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5f3b47b5-c32f-429f-b1ca-525f93117f37" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:56:52.692354 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.692303 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5f3b47b5-c32f-429f-b1ca-525f93117f37" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:56:52.692421 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.692398 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5f3b47b5-c32f-429f-b1ca-525f93117f37" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:56:52.692458 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.692406 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3b47b5-c32f-429f-b1ca-525f93117f37-kube-api-access-tjlpq" (OuterVolumeSpecName: "kube-api-access-tjlpq") pod "5f3b47b5-c32f-429f-b1ca-525f93117f37" (UID: "5f3b47b5-c32f-429f-b1ca-525f93117f37"). InnerVolumeSpecName "kube-api-access-tjlpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:56:52.790441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.790384 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3b47b5-c32f-429f-b1ca-525f93117f37-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:56:52.790441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.790405 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:56:52.790441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.790414 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjlpq\" (UniqueName: \"kubernetes.io/projected/5f3b47b5-c32f-429f-b1ca-525f93117f37-kube-api-access-tjlpq\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:56:52.790441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.790422 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:56:52.790441 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.790432 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:56:52.790649 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:52.790457 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5f3b47b5-c32f-429f-b1ca-525f93117f37-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 11:56:53.618259 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:53.618231 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj" Sep 11 11:56:53.640749 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:53.640662 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj"] Sep 11 11:56:53.644273 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:53.644243 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-l7jmj"] Sep 11 11:56:53.901004 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:56:53.900936 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" path="/var/lib/kubelet/pods/5f3b47b5-c32f-429f-b1ca-525f93117f37/volumes" Sep 11 11:57:47.078058 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078026 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx"] Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078338 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="feast-apply" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078349 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="feast-apply" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078362 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078367 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078374 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="feast-init" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078380 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="feast-init" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078385 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="registry" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078390 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="registry" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078395 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="offline" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078400 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="offline" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078410 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078414 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078456 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="registry" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078462 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="online" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078470 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="offline" Sep 11 11:57:47.078495 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.078476 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b47b5-c32f-429f-b1ca-525f93117f37" containerName="ui" Sep 11 11:57:47.081780 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.081762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.084614 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.084594 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 11 11:57:47.085000 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.084984 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 11 11:57:47.085179 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.085163 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 11 11:57:47.085957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.085933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 11 11:57:47.085957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.085949 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-qlxs7\"" Sep 11 11:57:47.085957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.085956 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 11 11:57:47.086211 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.085993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 11 11:57:47.096650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.096632 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx"] Sep 11 11:57:47.269792 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.269765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.269957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.269797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.269957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.269831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3577c28e-8d08-4060-9b91-aebd4e207d32-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.269957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.269904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.269957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.269927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3577c28e-8d08-4060-9b91-aebd4e207d32-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.269957 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.269955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trwgf\" (UniqueName: \"kubernetes.io/projected/3577c28e-8d08-4060-9b91-aebd4e207d32-kube-api-access-trwgf\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.370561 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.370532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.370561 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.370561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.370782 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.370582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3577c28e-8d08-4060-9b91-aebd4e207d32-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.370782 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.370617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.370782 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.370642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3577c28e-8d08-4060-9b91-aebd4e207d32-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.370782 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.370671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trwgf\" (UniqueName: \"kubernetes.io/projected/3577c28e-8d08-4060-9b91-aebd4e207d32-kube-api-access-trwgf\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.370782 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:57:47.370689 2579 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 11 11:57:47.371073 ip-10-0-142-17 kubenswrapper[2579]: E0911 11:57:47.370787 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls podName:3577c28e-8d08-4060-9b91-aebd4e207d32 nodeName:}" failed. No retries permitted until 2026-09-11 11:57:47.870765268 +0000 UTC m=+1644.406672246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" (UID: "3577c28e-8d08-4060-9b91-aebd4e207d32") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 11 11:57:47.371131 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.371114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3577c28e-8d08-4060-9b91-aebd4e207d32-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.371372 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.371351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3577c28e-8d08-4060-9b91-aebd4e207d32-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.373254 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.373234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.373304 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.373281 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.379705 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.379680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trwgf\" (UniqueName: \"kubernetes.io/projected/3577c28e-8d08-4060-9b91-aebd4e207d32-kube-api-access-trwgf\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.875841 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.875815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.878283 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.878253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-hnqxx\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:47.991375 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:47.991343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:57:48.266807 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:48.266782 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx"] Sep 11 11:57:48.268340 ip-10-0-142-17 kubenswrapper[2579]: W0911 11:57:48.268313 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3577c28e_8d08_4060_9b91_aebd4e207d32.slice/crio-aa561a4add62f64f8953e51b071a88937b1491ac01b4e754d5dbd648e3223e4e WatchSource:0}: Error finding container aa561a4add62f64f8953e51b071a88937b1491ac01b4e754d5dbd648e3223e4e: Status 404 returned error can't find the container with id aa561a4add62f64f8953e51b071a88937b1491ac01b4e754d5dbd648e3223e4e Sep 11 11:57:48.825409 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:48.825365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerStarted","Data":"fc0fc7e08b869fbabef59f2ded249328870470e656eacc032330bf572f58be5a"} Sep 11 11:57:48.825579 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:48.825421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerStarted","Data":"aa561a4add62f64f8953e51b071a88937b1491ac01b4e754d5dbd648e3223e4e"} Sep 11 11:57:53.845040 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:53.845006 2579 generic.go:356] "Generic (PLEG): container finished" podID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerID="fc0fc7e08b869fbabef59f2ded249328870470e656eacc032330bf572f58be5a" exitCode=0 Sep 11 11:57:53.845500 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:53.845084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerDied","Data":"fc0fc7e08b869fbabef59f2ded249328870470e656eacc032330bf572f58be5a"} Sep 11 11:57:54.850216 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:57:54.850183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerStarted","Data":"69f2de3a1444a9f78ccf2a4c75584b8c5bc1f9047c51e93d4fe0e306bf025d7a"} Sep 11 11:58:17.941494 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:17.941456 2579 generic.go:356] "Generic (PLEG): container finished" podID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerID="69f2de3a1444a9f78ccf2a4c75584b8c5bc1f9047c51e93d4fe0e306bf025d7a" exitCode=0 Sep 11 11:58:17.941936 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:17.941530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerDied","Data":"69f2de3a1444a9f78ccf2a4c75584b8c5bc1f9047c51e93d4fe0e306bf025d7a"} Sep 11 11:58:18.950516 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:18.950475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerStarted","Data":"5c2627f21672da70c13f96ba0c2a31f5a9194c4305a0945cbf445e25c49e6bf9"} Sep 11 11:58:18.951075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:18.950526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerStarted","Data":"1b501faf263045ca748aa229a7837b80c047135b56b2599e029bdd0ff3bc5557"} Sep 11 11:58:18.951075 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:18.950544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerStarted","Data":"c63bd5d760812d6e6577da98646b60c098d3ba0865081221824be2e15ae469ff"} Sep 11 11:58:20.991851 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:20.991802 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:20.992355 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:20.991868 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:20.992355 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:20.991887 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:20.993442 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:20.993398 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused" Sep 11 11:58:20.993442 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:20.993428 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 11 11:58:20.993650 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:20.993446 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 11 11:58:23.991687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:23.991637 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 11 11:58:23.991687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:23.991649 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused" Sep 11 11:58:23.991687 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:23.991665 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 11 11:58:26.992999 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:26.992965 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:26.993448 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:26.993129 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:26.993591 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:26.993573 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:26.993682 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:26.993607 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:26.993829 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:26.993813 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:26.993931 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:26.993915 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:27.027206 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:27.027163 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podStartSLOduration=40.027146984 podStartE2EDuration="40.027146984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:18.99124705 +0000 UTC m=+1675.527154049" watchObservedRunningTime="2026-09-11 11:58:27.027146984 +0000 UTC m=+1683.563053983" Sep 11 11:58:27.992021 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:27.991986 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:27.992515 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:27.992482 2579 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 11 11:58:30.858637 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:30.858606 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:58:31.765319 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:58:31.765288 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:59:08.995864 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:08.995818 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 11:59:21.863865 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:21.863816 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 11:59:28.995850 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:28.995809 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 11 11:59:31.863873 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:31.863831 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 11:59:33.240698 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.240664 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx"] Sep 11 11:59:33.241134 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.240885 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" containerID="cri-o://c63bd5d760812d6e6577da98646b60c098d3ba0865081221824be2e15ae469ff" gracePeriod=30 Sep 11 11:59:33.241134 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.240964 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" containerID="cri-o://5c2627f21672da70c13f96ba0c2a31f5a9194c4305a0945cbf445e25c49e6bf9" gracePeriod=30 Sep 11 11:59:33.241134 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.240973 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="offline" containerID="cri-o://1b501faf263045ca748aa229a7837b80c047135b56b2599e029bdd0ff3bc5557" gracePeriod=30 Sep 11 11:59:33.443279 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.443242 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4"] Sep 11 11:59:33.443516 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.443483 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="registry" containerID="cri-o://94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0" gracePeriod=30 Sep 11 11:59:33.443630 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.443521 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" containerID="cri-o://7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6" gracePeriod=30 Sep 11 11:59:33.443630 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.443537 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="offline" containerID="cri-o://a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3" gracePeriod=30 Sep 11 11:59:33.443757 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:33.443546 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" containerID="cri-o://17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab" gracePeriod=30 Sep 11 11:59:34.271173 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.271138 2579 generic.go:356] "Generic (PLEG): container finished" podID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerID="5c2627f21672da70c13f96ba0c2a31f5a9194c4305a0945cbf445e25c49e6bf9" exitCode=0 Sep 11 11:59:34.271173 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.271163 2579 generic.go:356] "Generic (PLEG): container finished" podID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerID="c63bd5d760812d6e6577da98646b60c098d3ba0865081221824be2e15ae469ff" exitCode=0 Sep 11 11:59:34.271518 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.271211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerDied","Data":"5c2627f21672da70c13f96ba0c2a31f5a9194c4305a0945cbf445e25c49e6bf9"} Sep 11 11:59:34.271518 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.271258 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerDied","Data":"c63bd5d760812d6e6577da98646b60c098d3ba0865081221824be2e15ae469ff"} Sep 11 11:59:34.274004 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.273982 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerID="7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6" exitCode=0 Sep 11 11:59:34.274004 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.274003 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerID="17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab" exitCode=0 Sep 11 11:59:34.274149 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.274022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerDied","Data":"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6"} Sep 11 11:59:34.274149 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.274048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerDied","Data":"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab"} Sep 11 11:59:34.583291 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.583245 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 11 11:59:34.583291 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:34.583281 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 11 11:59:36.994085 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:36.994037 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 11 11:59:40.859608 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:40.859565 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 11 11:59:40.860110 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:40.859671 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 11:59:44.583429 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:44.583384 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 11 11:59:44.583907 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:44.583429 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 11 11:59:46.994109 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:46.994066 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 11 11:59:50.859670 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:50.859630 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 11 11:59:54.582712 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:54.582666 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 11 11:59:54.583296 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:54.582666 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 11 11:59:54.583296 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:54.582859 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:59:54.583296 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:54.582930 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 11:59:56.993792 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:56.993750 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 11 11:59:56.994176 ip-10-0-142-17 kubenswrapper[2579]: I0911 11:59:56.993882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 12:00:00.859425 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:00.859384 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 11 12:00:03.387964 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:03.387928 2579 generic.go:356] "Generic (PLEG): container finished" podID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerID="1b501faf263045ca748aa229a7837b80c047135b56b2599e029bdd0ff3bc5557" exitCode=137 Sep 11 12:00:03.388323 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:03.387994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerDied","Data":"1b501faf263045ca748aa229a7837b80c047135b56b2599e029bdd0ff3bc5557"} Sep 11 12:00:03.881899 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:03.881871 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 12:00:04.021145 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.021121 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-online-tls\") pod \"3577c28e-8d08-4060-9b91-aebd4e207d32\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " Sep 11 12:00:04.021278 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.021158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trwgf\" (UniqueName: \"kubernetes.io/projected/3577c28e-8d08-4060-9b91-aebd4e207d32-kube-api-access-trwgf\") pod \"3577c28e-8d08-4060-9b91-aebd4e207d32\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " Sep 11 12:00:04.021278 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.021204 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls\") pod \"3577c28e-8d08-4060-9b91-aebd4e207d32\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " Sep 11 12:00:04.021278 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.021238 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-offline-tls\") pod \"3577c28e-8d08-4060-9b91-aebd4e207d32\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " Sep 11 12:00:04.021427 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.021290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3577c28e-8d08-4060-9b91-aebd4e207d32-feast-data\") pod \"3577c28e-8d08-4060-9b91-aebd4e207d32\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " Sep 11 12:00:04.021427 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.021331 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3577c28e-8d08-4060-9b91-aebd4e207d32-registry-tls\") pod \"3577c28e-8d08-4060-9b91-aebd4e207d32\" (UID: \"3577c28e-8d08-4060-9b91-aebd4e207d32\") " Sep 11 12:00:04.021947 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.021864 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3577c28e-8d08-4060-9b91-aebd4e207d32-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3577c28e-8d08-4060-9b91-aebd4e207d32" (UID: "3577c28e-8d08-4060-9b91-aebd4e207d32"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:00:04.022180 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.022155 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/3577c28e-8d08-4060-9b91-aebd4e207d32-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.022578 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.022552 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3577c28e-8d08-4060-9b91-aebd4e207d32-feast-data" (OuterVolumeSpecName: "feast-data") pod "3577c28e-8d08-4060-9b91-aebd4e207d32" (UID: "3577c28e-8d08-4060-9b91-aebd4e207d32"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:04.024581 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.024536 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "3577c28e-8d08-4060-9b91-aebd4e207d32" (UID: "3577c28e-8d08-4060-9b91-aebd4e207d32"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:04.024686 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.024579 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3577c28e-8d08-4060-9b91-aebd4e207d32-kube-api-access-trwgf" (OuterVolumeSpecName: "kube-api-access-trwgf") pod "3577c28e-8d08-4060-9b91-aebd4e207d32" (UID: "3577c28e-8d08-4060-9b91-aebd4e207d32"). InnerVolumeSpecName "kube-api-access-trwgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:00:04.024686 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.024597 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-online-tls" (OuterVolumeSpecName: "online-tls") pod "3577c28e-8d08-4060-9b91-aebd4e207d32" (UID: "3577c28e-8d08-4060-9b91-aebd4e207d32"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:04.024686 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.024608 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "3577c28e-8d08-4060-9b91-aebd4e207d32" (UID: "3577c28e-8d08-4060-9b91-aebd4e207d32"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:04.089124 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.089100 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 12:00:04.122832 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.122804 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.122832 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.122827 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trwgf\" (UniqueName: \"kubernetes.io/projected/3577c28e-8d08-4060-9b91-aebd4e207d32-kube-api-access-trwgf\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.122832 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.122844 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.123033 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.122853 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/3577c28e-8d08-4060-9b91-aebd4e207d32-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.123033 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.122862 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/3577c28e-8d08-4060-9b91-aebd4e207d32-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.223858 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.223829 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls\") pod \"4d5e5f6d-af5b-497f-b952-0f1db866398d\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " Sep 11 12:00:04.223858 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.223856 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-registry-tls\") pod \"4d5e5f6d-af5b-497f-b952-0f1db866398d\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " Sep 11 12:00:04.224045 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.223880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5e5f6d-af5b-497f-b952-0f1db866398d-feast-data\") pod \"4d5e5f6d-af5b-497f-b952-0f1db866398d\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " Sep 11 12:00:04.224045 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.223910 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-online-tls\") pod \"4d5e5f6d-af5b-497f-b952-0f1db866398d\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " Sep 11 12:00:04.224045 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.223952 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpnnr\" (UniqueName: \"kubernetes.io/projected/4d5e5f6d-af5b-497f-b952-0f1db866398d-kube-api-access-fpnnr\") pod \"4d5e5f6d-af5b-497f-b952-0f1db866398d\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " Sep 11 12:00:04.224045 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.223970 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-offline-tls\") pod \"4d5e5f6d-af5b-497f-b952-0f1db866398d\" (UID: \"4d5e5f6d-af5b-497f-b952-0f1db866398d\") " Sep 11 12:00:04.224437 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.224399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5e5f6d-af5b-497f-b952-0f1db866398d-feast-data" (OuterVolumeSpecName: "feast-data") pod "4d5e5f6d-af5b-497f-b952-0f1db866398d" (UID: "4d5e5f6d-af5b-497f-b952-0f1db866398d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:04.226249 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.226219 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5e5f6d-af5b-497f-b952-0f1db866398d-kube-api-access-fpnnr" (OuterVolumeSpecName: "kube-api-access-fpnnr") pod "4d5e5f6d-af5b-497f-b952-0f1db866398d" (UID: "4d5e5f6d-af5b-497f-b952-0f1db866398d"). InnerVolumeSpecName "kube-api-access-fpnnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:00:04.226372 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.226278 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4d5e5f6d-af5b-497f-b952-0f1db866398d" (UID: "4d5e5f6d-af5b-497f-b952-0f1db866398d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:04.226372 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.226361 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-online-tls" (OuterVolumeSpecName: "online-tls") pod "4d5e5f6d-af5b-497f-b952-0f1db866398d" (UID: "4d5e5f6d-af5b-497f-b952-0f1db866398d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:04.226531 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.226515 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4d5e5f6d-af5b-497f-b952-0f1db866398d" (UID: "4d5e5f6d-af5b-497f-b952-0f1db866398d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:04.226572 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.226531 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "4d5e5f6d-af5b-497f-b952-0f1db866398d" (UID: "4d5e5f6d-af5b-497f-b952-0f1db866398d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:04.324506 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.324433 2579 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-ui-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.324506 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.324455 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-registry-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.324506 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.324465 2579 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4d5e5f6d-af5b-497f-b952-0f1db866398d-feast-data\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.324506 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.324473 2579 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-online-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.324506 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.324480 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpnnr\" (UniqueName: \"kubernetes.io/projected/4d5e5f6d-af5b-497f-b952-0f1db866398d-kube-api-access-fpnnr\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.324506 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.324488 2579 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4d5e5f6d-af5b-497f-b952-0f1db866398d-offline-tls\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:04.394060 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.394029 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" Sep 11 12:00:04.394483 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.394053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx" event={"ID":"3577c28e-8d08-4060-9b91-aebd4e207d32","Type":"ContainerDied","Data":"aa561a4add62f64f8953e51b071a88937b1491ac01b4e754d5dbd648e3223e4e"} Sep 11 12:00:04.394483 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.394114 2579 scope.go:117] "RemoveContainer" containerID="5c2627f21672da70c13f96ba0c2a31f5a9194c4305a0945cbf445e25c49e6bf9" Sep 11 12:00:04.397428 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.397403 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerID="a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3" exitCode=137 Sep 11 12:00:04.397553 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.397429 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerID="94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0" exitCode=137 Sep 11 12:00:04.397553 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.397470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerDied","Data":"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3"} Sep 11 12:00:04.397553 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.397518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerDied","Data":"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0"} Sep 11 12:00:04.397553 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.397531 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" Sep 11 12:00:04.397553 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.397537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4" event={"ID":"4d5e5f6d-af5b-497f-b952-0f1db866398d","Type":"ContainerDied","Data":"f7065e0f4b3033ea3be47b312888e67305e02602848674adb8ba29e4e06bbeae"} Sep 11 12:00:04.403908 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.403892 2579 scope.go:117] "RemoveContainer" containerID="1b501faf263045ca748aa229a7837b80c047135b56b2599e029bdd0ff3bc5557" Sep 11 12:00:04.412351 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.412336 2579 scope.go:117] "RemoveContainer" containerID="c63bd5d760812d6e6577da98646b60c098d3ba0865081221824be2e15ae469ff" Sep 11 12:00:04.419501 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.419484 2579 scope.go:117] "RemoveContainer" containerID="69f2de3a1444a9f78ccf2a4c75584b8c5bc1f9047c51e93d4fe0e306bf025d7a" Sep 11 12:00:04.424762 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.424739 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx"] Sep 11 12:00:04.427486 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.427471 2579 scope.go:117] "RemoveContainer" containerID="fc0fc7e08b869fbabef59f2ded249328870470e656eacc032330bf572f58be5a" Sep 11 12:00:04.432820 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.432798 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-hnqxx"] Sep 11 12:00:04.434881 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.434804 2579 scope.go:117] "RemoveContainer" containerID="7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6" Sep 11 12:00:04.441823 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.441801 2579 scope.go:117] "RemoveContainer" containerID="a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3" Sep 11 12:00:04.449013 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.448992 2579 scope.go:117] "RemoveContainer" containerID="17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab" Sep 11 12:00:04.450440 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.450421 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4"] Sep 11 12:00:04.455543 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.455520 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-7wkm4"] Sep 11 12:00:04.456830 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.456812 2579 scope.go:117] "RemoveContainer" containerID="94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0" Sep 11 12:00:04.469772 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.469748 2579 scope.go:117] "RemoveContainer" containerID="db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7" Sep 11 12:00:04.477168 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.477145 2579 scope.go:117] "RemoveContainer" containerID="d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6" Sep 11 12:00:04.484211 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.484192 2579 scope.go:117] "RemoveContainer" containerID="7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6" Sep 11 12:00:04.484493 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:04.484475 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6\": container with ID starting with 7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6 not found: ID does not exist" containerID="7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6" Sep 11 12:00:04.484590 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.484506 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6"} err="failed to get container status \"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6\": rpc error: code = NotFound desc = could not find container \"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6\": container with ID starting with 7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6 not found: ID does not exist" Sep 11 12:00:04.484590 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.484531 2579 scope.go:117] "RemoveContainer" containerID="a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3" Sep 11 12:00:04.484834 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:04.484815 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3\": container with ID starting with a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3 not found: ID does not exist" containerID="a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3" Sep 11 12:00:04.484897 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.484840 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3"} err="failed to get container status \"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3\": rpc error: code = NotFound desc = could not find container \"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3\": container with ID starting with a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3 not found: ID does not exist" Sep 11 12:00:04.484897 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.484858 2579 scope.go:117] "RemoveContainer" containerID="17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab" Sep 11 12:00:04.485105 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:04.485085 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab\": container with ID starting with 17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab not found: ID does not exist" containerID="17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab" Sep 11 12:00:04.485171 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.485112 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab"} err="failed to get container status \"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab\": rpc error: code = NotFound desc = could not find container \"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab\": container with ID starting with 17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab not found: ID does not exist" Sep 11 12:00:04.485171 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.485133 2579 scope.go:117] "RemoveContainer" containerID="94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0" Sep 11 12:00:04.485390 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:04.485371 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0\": container with ID starting with 94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0 not found: ID does not exist" containerID="94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0" Sep 11 12:00:04.485432 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.485395 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0"} err="failed to get container status \"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0\": rpc error: code = NotFound desc = could not find container \"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0\": container with ID starting with 94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0 not found: ID does not exist" Sep 11 12:00:04.485432 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.485409 2579 scope.go:117] "RemoveContainer" containerID="db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7" Sep 11 12:00:04.485651 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:04.485635 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7\": container with ID starting with db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7 not found: ID does not exist" containerID="db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7" Sep 11 12:00:04.485694 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.485655 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7"} err="failed to get container status \"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7\": rpc error: code = NotFound desc = could not find container \"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7\": container with ID starting with db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7 not found: ID does not exist" Sep 11 12:00:04.485694 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.485669 2579 scope.go:117] "RemoveContainer" containerID="d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6" Sep 11 12:00:04.486007 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:04.485992 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6\": container with ID starting with d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6 not found: ID does not exist" containerID="d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6" Sep 11 12:00:04.486063 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486011 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6"} err="failed to get container status \"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6\": rpc error: code = NotFound desc = could not find container \"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6\": container with ID starting with d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6 not found: ID does not exist" Sep 11 12:00:04.486063 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486024 2579 scope.go:117] "RemoveContainer" containerID="7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6" Sep 11 12:00:04.486269 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486251 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6"} err="failed to get container status \"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6\": rpc error: code = NotFound desc = could not find container \"7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6\": container with ID starting with 7055d14dcccf7e3a2b6e9ed517dbd2b7416b671a99bfc41ff91300618a1347f6 not found: ID does not exist" Sep 11 12:00:04.486318 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486271 2579 scope.go:117] "RemoveContainer" containerID="a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3" Sep 11 12:00:04.486484 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486468 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3"} err="failed to get container status \"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3\": rpc error: code = NotFound desc = could not find container \"a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3\": container with ID starting with a1879d3e897a68da82aa9008b53d2ef9d16f6ffe8a75b2fa6a503079a4c285a3 not found: ID does not exist" Sep 11 12:00:04.486533 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486485 2579 scope.go:117] "RemoveContainer" containerID="17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab" Sep 11 12:00:04.486740 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486696 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab"} err="failed to get container status \"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab\": rpc error: code = NotFound desc = could not find container \"17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab\": container with ID starting with 17195a2e2a1f386d434cd55e91dc01de83c348c9a449b0a654cdbed64dc90aab not found: ID does not exist" Sep 11 12:00:04.486809 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486748 2579 scope.go:117] "RemoveContainer" containerID="94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0" Sep 11 12:00:04.487006 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.486989 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0"} err="failed to get container status \"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0\": rpc error: code = NotFound desc = could not find container \"94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0\": container with ID starting with 94d71fb6db1ebb5faac033ecc141195c1aef74f817b6e952833cb7482cd6ffa0 not found: ID does not exist" Sep 11 12:00:04.487057 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.487007 2579 scope.go:117] "RemoveContainer" containerID="db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7" Sep 11 12:00:04.487229 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.487212 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7"} err="failed to get container status \"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7\": rpc error: code = NotFound desc = could not find container \"db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7\": container with ID starting with db8d169a7c7dd80aab3c0799da24b97c08bfba9a8c707ebe8a0045a071c9bbd7 not found: ID does not exist" Sep 11 12:00:04.487270 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.487230 2579 scope.go:117] "RemoveContainer" containerID="d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6" Sep 11 12:00:04.487440 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:04.487423 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6"} err="failed to get container status \"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6\": rpc error: code = NotFound desc = could not find container \"d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6\": container with ID starting with d9e37d8b9f1146b6949e5d846a5e77405fcc1172c7c2bd13bc583864730731a6 not found: ID does not exist" Sep 11 12:00:05.899356 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:05.899325 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" path="/var/lib/kubelet/pods/3577c28e-8d08-4060-9b91-aebd4e207d32/volumes" Sep 11 12:00:05.899852 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:05.899836 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" path="/var/lib/kubelet/pods/4d5e5f6d-af5b-497f-b952-0f1db866398d/volumes" Sep 11 12:00:22.909183 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909148 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-spr2p/must-gather-jhjgd"] Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909409 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="offline" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909419 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="offline" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909433 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="registry" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909438 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="registry" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909446 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="feast-apply" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909452 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="feast-apply" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909458 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909464 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909471 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="feast-init" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909476 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="feast-init" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909482 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="offline" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909487 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="offline" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909495 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909499 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909507 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="feast-init" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909512 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="feast-init" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909520 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909524 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909530 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909535 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909541 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="feast-apply" Sep 11 12:00:22.909565 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909546 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="feast-apply" Sep 11 12:00:22.910251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909584 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="ui" Sep 11 12:00:22.910251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909591 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="online" Sep 11 12:00:22.910251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909596 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="ui" Sep 11 12:00:22.910251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909603 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="registry" Sep 11 12:00:22.910251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909609 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="offline" Sep 11 12:00:22.910251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909614 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3577c28e-8d08-4060-9b91-aebd4e207d32" containerName="online" Sep 11 12:00:22.910251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.909620 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5e5f6d-af5b-497f-b952-0f1db866398d" containerName="offline" Sep 11 12:00:22.912566 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.912549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:22.921813 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.921784 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-spr2p\"/\"default-dockercfg-p6rps\"" Sep 11 12:00:22.921969 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.921844 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spr2p\"/\"kube-root-ca.crt\"" Sep 11 12:00:22.922334 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.922318 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spr2p\"/\"openshift-service-ca.crt\"" Sep 11 12:00:22.949165 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.949139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-must-gather-output\") pod \"must-gather-jhjgd\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:22.949254 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.949176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4b6f\" (UniqueName: \"kubernetes.io/projected/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-kube-api-access-k4b6f\") pod \"must-gather-jhjgd\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:22.989042 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:22.989017 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spr2p/must-gather-jhjgd"] Sep 11 12:00:23.050386 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:23.050353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4b6f\" (UniqueName: \"kubernetes.io/projected/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-kube-api-access-k4b6f\") pod \"must-gather-jhjgd\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:23.050505 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:23.050445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-must-gather-output\") pod \"must-gather-jhjgd\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:23.050741 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:23.050709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-must-gather-output\") pod \"must-gather-jhjgd\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:23.066107 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:23.066079 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4b6f\" (UniqueName: \"kubernetes.io/projected/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-kube-api-access-k4b6f\") pod \"must-gather-jhjgd\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:23.221445 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:23.221363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:23.495215 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:23.495122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spr2p/must-gather-jhjgd"] Sep 11 12:00:23.499088 ip-10-0-142-17 kubenswrapper[2579]: W0911 12:00:23.499063 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92b7c85c_1e16_400d_8adc_c5ea14f82fb5.slice/crio-c5c998184e0e90ac461dadec351c1756981c29c4a2eb3d6a286eaf8f51597c5a WatchSource:0}: Error finding container c5c998184e0e90ac461dadec351c1756981c29c4a2eb3d6a286eaf8f51597c5a: Status 404 returned error can't find the container with id c5c998184e0e90ac461dadec351c1756981c29c4a2eb3d6a286eaf8f51597c5a Sep 11 12:00:23.501195 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:23.501181 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 12:00:24.471728 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:24.471680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spr2p/must-gather-jhjgd" event={"ID":"92b7c85c-1e16-400d-8adc-c5ea14f82fb5","Type":"ContainerStarted","Data":"c5c998184e0e90ac461dadec351c1756981c29c4a2eb3d6a286eaf8f51597c5a"} Sep 11 12:00:31.503411 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:31.503320 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spr2p/must-gather-jhjgd" event={"ID":"92b7c85c-1e16-400d-8adc-c5ea14f82fb5","Type":"ContainerStarted","Data":"2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376"} Sep 11 12:00:31.503411 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:31.503376 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spr2p/must-gather-jhjgd" event={"ID":"92b7c85c-1e16-400d-8adc-c5ea14f82fb5","Type":"ContainerStarted","Data":"a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79"} Sep 11 12:00:31.531737 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:31.531655 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-spr2p/must-gather-jhjgd" podStartSLOduration=1.928996978 podStartE2EDuration="9.531623207s" totalImagesPullingTime="7.602626229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:00:31.530618363 +0000 UTC m=+1808.066525360" watchObservedRunningTime="2026-09-11 12:00:31.531623207 +0000 UTC m=+1808.067530205" Sep 11 12:00:41.543387 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:41.543354 2579 generic.go:356] "Generic (PLEG): container finished" podID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerID="a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79" exitCode=0 Sep 11 12:00:41.543814 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:41.543431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spr2p/must-gather-jhjgd" event={"ID":"92b7c85c-1e16-400d-8adc-c5ea14f82fb5","Type":"ContainerDied","Data":"a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79"} Sep 11 12:00:41.543814 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:41.543749 2579 scope.go:117] "RemoveContainer" containerID="a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79" Sep 11 12:00:41.791638 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:41.791611 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-spr2p_must-gather-jhjgd_92b7c85c-1e16-400d-8adc-c5ea14f82fb5/gather/0.log" Sep 11 12:00:42.335503 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.335420 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-54pxb/must-gather-dcjf7"] Sep 11 12:00:42.338225 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.338209 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.347279 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.347258 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-54pxb\"/\"kube-root-ca.crt\"" Sep 11 12:00:42.348604 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.348588 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-54pxb\"/\"default-dockercfg-xqgv4\"" Sep 11 12:00:42.348702 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.348588 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-54pxb\"/\"openshift-service-ca.crt\"" Sep 11 12:00:42.354128 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.354109 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54pxb/must-gather-dcjf7"] Sep 11 12:00:42.407146 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.407110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdk6\" (UniqueName: \"kubernetes.io/projected/d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab-kube-api-access-nqdk6\") pod \"must-gather-dcjf7\" (UID: \"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab\") " pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.407351 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.407178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab-must-gather-output\") pod \"must-gather-dcjf7\" (UID: \"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab\") " pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.508367 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.508320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab-must-gather-output\") pod \"must-gather-dcjf7\" (UID: \"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab\") " pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.508367 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.508380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdk6\" (UniqueName: \"kubernetes.io/projected/d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab-kube-api-access-nqdk6\") pod \"must-gather-dcjf7\" (UID: \"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab\") " pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.508741 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.508700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab-must-gather-output\") pod \"must-gather-dcjf7\" (UID: \"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab\") " pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.517994 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.517976 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdk6\" (UniqueName: \"kubernetes.io/projected/d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab-kube-api-access-nqdk6\") pod \"must-gather-dcjf7\" (UID: \"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab\") " pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.647412 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.647385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54pxb/must-gather-dcjf7" Sep 11 12:00:42.917020 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:42.916939 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54pxb/must-gather-dcjf7"] Sep 11 12:00:42.919676 ip-10-0-142-17 kubenswrapper[2579]: W0911 12:00:42.919643 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2040edc_5f62_4bc4_bf9c_1c9e4d7d28ab.slice/crio-0627f5a7fdf130c8daa4becd349ff24844fb533d85e38293bfe5c9cd3213e0ea WatchSource:0}: Error finding container 0627f5a7fdf130c8daa4becd349ff24844fb533d85e38293bfe5c9cd3213e0ea: Status 404 returned error can't find the container with id 0627f5a7fdf130c8daa4becd349ff24844fb533d85e38293bfe5c9cd3213e0ea Sep 11 12:00:43.550663 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:43.550627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54pxb/must-gather-dcjf7" event={"ID":"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab","Type":"ContainerStarted","Data":"0627f5a7fdf130c8daa4becd349ff24844fb533d85e38293bfe5c9cd3213e0ea"} Sep 11 12:00:44.559042 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:44.559004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54pxb/must-gather-dcjf7" event={"ID":"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab","Type":"ContainerStarted","Data":"8fc62aec0593ea769a9c7f207ff08d80b6492d645ccdb271c2fe9501b75f6c63"} Sep 11 12:00:44.559589 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:44.559548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54pxb/must-gather-dcjf7" event={"ID":"d2040edc-5f62-4bc4-bf9c-1c9e4d7d28ab","Type":"ContainerStarted","Data":"103e268aa69a5edc24f65e4453e6ed96876c8543af5118f6591ff60ee5100e9e"} Sep 11 12:00:44.580566 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:44.580510 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-54pxb/must-gather-dcjf7" podStartSLOduration=1.652082117 podStartE2EDuration="2.580488953s" totalImagesPullingTime="928.406836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:00:44.577388034 +0000 UTC m=+1821.113295043" watchObservedRunningTime="2026-09-11 12:00:44.580488953 +0000 UTC m=+1821.116395951" Sep 11 12:00:45.472900 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:45.472868 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d6f7p_222b1b29-9ac9-4418-b6c2-271ff86e4850/global-pull-secret-syncer/0.log" Sep 11 12:00:45.635819 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:45.635786 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dxpvh_a0f63ea2-446e-4fdc-9da3-d28ac08c3727/konnectivity-agent/0.log" Sep 11 12:00:45.750629 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:45.750555 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-17.ec2.internal_c14bf53574a5faf7f92e569f4d21b76b/haproxy/0.log" Sep 11 12:00:47.155334 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.155293 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-spr2p/must-gather-jhjgd"] Sep 11 12:00:47.155951 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.155546 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-spr2p/must-gather-jhjgd" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerName="copy" containerID="cri-o://2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376" gracePeriod=2 Sep 11 12:00:47.162137 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.162115 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-spr2p/must-gather-jhjgd"] Sep 11 12:00:47.530744 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.530091 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-spr2p_must-gather-jhjgd_92b7c85c-1e16-400d-8adc-c5ea14f82fb5/copy/0.log" Sep 11 12:00:47.538313 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.536868 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:47.542595 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:47.542560 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jhjgd\" is forbidden: User \"system:node:ip-10-0-142-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-spr2p\": no relationship found between node 'ip-10-0-142-17.ec2.internal' and this object" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:47.580743 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.580110 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-spr2p_must-gather-jhjgd_92b7c85c-1e16-400d-8adc-c5ea14f82fb5/copy/0.log" Sep 11 12:00:47.580743 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.580611 2579 generic.go:356] "Generic (PLEG): container finished" podID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerID="2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376" exitCode=143 Sep 11 12:00:47.580991 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.580759 2579 scope.go:117] "RemoveContainer" containerID="2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376" Sep 11 12:00:47.580991 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.580932 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:47.583962 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:47.583941 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jhjgd\" is forbidden: User \"system:node:ip-10-0-142-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-spr2p\": no relationship found between node 'ip-10-0-142-17.ec2.internal' and this object" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" pod="openshift-must-gather-spr2p/must-gather-jhjgd" Sep 11 12:00:47.605889 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.605859 2579 scope.go:117] "RemoveContainer" containerID="a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79" Sep 11 12:00:47.634057 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.633993 2579 scope.go:117] "RemoveContainer" containerID="2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376" Sep 11 12:00:47.634875 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:47.634468 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376\": container with ID starting with 2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376 not found: ID does not exist" containerID="2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376" Sep 11 12:00:47.634875 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.634506 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376"} err="failed to get container status \"2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376\": rpc error: code = NotFound desc = could not find container \"2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376\": container with ID starting with 2dc4e9c40f64a7cb8212202b046fb14382d6971dbc64787ecce408b70caf9376 not found: ID does not exist" Sep 11 12:00:47.634875 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.634557 2579 scope.go:117] "RemoveContainer" containerID="a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79" Sep 11 12:00:47.635114 ip-10-0-142-17 kubenswrapper[2579]: E0911 12:00:47.634997 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79\": container with ID starting with a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79 not found: ID does not exist" containerID="a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79" Sep 11 12:00:47.635114 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.635032 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79"} err="failed to get container status \"a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79\": rpc error: code = NotFound desc = could not find container \"a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79\": container with ID starting with a7f4fc594e570b02dbb8128ca65641a24db8a19c6c214dbf496847619870fb79 not found: ID does not exist" Sep 11 12:00:47.653659 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.653634 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4b6f\" (UniqueName: \"kubernetes.io/projected/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-kube-api-access-k4b6f\") pod \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " Sep 11 12:00:47.654701 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.654649 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "92b7c85c-1e16-400d-8adc-c5ea14f82fb5" (UID: "92b7c85c-1e16-400d-8adc-c5ea14f82fb5"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:47.654992 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.654977 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-must-gather-output\") pod \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\" (UID: \"92b7c85c-1e16-400d-8adc-c5ea14f82fb5\") " Sep 11 12:00:47.655359 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.655344 2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-must-gather-output\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:47.660434 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.660412 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-kube-api-access-k4b6f" (OuterVolumeSpecName: "kube-api-access-k4b6f") pod "92b7c85c-1e16-400d-8adc-c5ea14f82fb5" (UID: "92b7c85c-1e16-400d-8adc-c5ea14f82fb5"). InnerVolumeSpecName "kube-api-access-k4b6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:00:47.756487 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.756382 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4b6f\" (UniqueName: \"kubernetes.io/projected/92b7c85c-1e16-400d-8adc-c5ea14f82fb5-kube-api-access-k4b6f\") on node \"ip-10-0-142-17.ec2.internal\" DevicePath \"\"" Sep 11 12:00:47.907276 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:47.907239 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" path="/var/lib/kubelet/pods/92b7c85c-1e16-400d-8adc-c5ea14f82fb5/volumes" Sep 11 12:00:49.081035 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.081006 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6aa8c45-ce35-499a-8514-3c8265a9472b/alertmanager/0.log" Sep 11 12:00:49.137300 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.137266 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6aa8c45-ce35-499a-8514-3c8265a9472b/config-reloader/0.log" Sep 11 12:00:49.175409 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.175369 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6aa8c45-ce35-499a-8514-3c8265a9472b/kube-rbac-proxy-web/0.log" Sep 11 12:00:49.215041 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.215010 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6aa8c45-ce35-499a-8514-3c8265a9472b/kube-rbac-proxy/0.log" Sep 11 12:00:49.266870 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.266840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6aa8c45-ce35-499a-8514-3c8265a9472b/kube-rbac-proxy-metric/0.log" Sep 11 12:00:49.335082 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.334908 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6aa8c45-ce35-499a-8514-3c8265a9472b/prom-label-proxy/0.log" Sep 11 12:00:49.371327 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.371294 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f6aa8c45-ce35-499a-8514-3c8265a9472b/init-config-reloader/0.log" Sep 11 12:00:49.455812 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.455784 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8js97_3f2a1a95-d84c-42e0-8a94-a0571411210b/kube-state-metrics/0.log" Sep 11 12:00:49.487305 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.487270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8js97_3f2a1a95-d84c-42e0-8a94-a0571411210b/kube-rbac-proxy-main/0.log" Sep 11 12:00:49.521229 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.521099 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-8js97_3f2a1a95-d84c-42e0-8a94-a0571411210b/kube-rbac-proxy-self/0.log" Sep 11 12:00:49.562575 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.562539 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cf9d5c9b8-zsvg2_59b5b439-d6c7-403c-928a-3815511de21a/metrics-server/0.log" Sep 11 12:00:49.614368 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.614334 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-j2d2q_a0d26acc-7d8f-4bd4-88c2-7f9f39eff74a/monitoring-plugin/0.log" Sep 11 12:00:49.791621 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.791594 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2qz5_4c4f2da0-ee51-463b-8913-b63d884aeec0/node-exporter/0.log" Sep 11 12:00:49.840562 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.840511 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2qz5_4c4f2da0-ee51-463b-8913-b63d884aeec0/kube-rbac-proxy/0.log" Sep 11 12:00:49.912562 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:49.912475 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2qz5_4c4f2da0-ee51-463b-8913-b63d884aeec0/init-textfile/0.log" Sep 11 12:00:50.459763 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:50.459677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-ntdz9_851078be-7141-4281-87c1-7d78c25b3ab4/prometheus-operator-admission-webhook/0.log" Sep 11 12:00:52.742856 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.742811 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln"] Sep 11 12:00:52.744038 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.744000 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerName="copy" Sep 11 12:00:52.744213 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.744199 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerName="copy" Sep 11 12:00:52.744336 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.744324 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerName="gather" Sep 11 12:00:52.744432 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.744420 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerName="gather" Sep 11 12:00:52.744582 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.744568 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerName="copy" Sep 11 12:00:52.744685 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.744674 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b7c85c-1e16-400d-8adc-c5ea14f82fb5" containerName="gather" Sep 11 12:00:52.750146 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.750125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:52.762409 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.762385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln"] Sep 11 12:00:52.854418 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.854389 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-mhwfg_0f79254d-a7c1-40ce-a08c-678276ff68ac/download-server/0.log" Sep 11 12:00:52.902630 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.902590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-proc\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:52.903019 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.902985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-sys\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:52.903189 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.903169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmzd\" (UniqueName: \"kubernetes.io/projected/240acc86-45eb-4e34-aa6d-1884e9699cce-kube-api-access-5jmzd\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:52.903354 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.903336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-lib-modules\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:52.903519 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:52.903500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-podres\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004339 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-sys\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004339 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmzd\" (UniqueName: \"kubernetes.io/projected/240acc86-45eb-4e34-aa6d-1884e9699cce-kube-api-access-5jmzd\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004339 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-lib-modules\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004339 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-podres\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004637 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-proc\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004637 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-sys\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004637 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004454 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-proc\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004637 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-lib-modules\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.004637 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.004511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/240acc86-45eb-4e34-aa6d-1884e9699cce-podres\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.023345 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.023318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmzd\" (UniqueName: \"kubernetes.io/projected/240acc86-45eb-4e34-aa6d-1884e9699cce-kube-api-access-5jmzd\") pod \"perf-node-gather-daemonset-9wwln\" (UID: \"240acc86-45eb-4e34-aa6d-1884e9699cce\") " pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.060169 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.060141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:53.360509 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.360475 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln"] Sep 11 12:00:53.614417 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.614331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" event={"ID":"240acc86-45eb-4e34-aa6d-1884e9699cce","Type":"ContainerStarted","Data":"2d73b088b6a4b12aa0ebdba73cf95f05b8ff8c980d0b7cc92abf83abee910dc3"} Sep 11 12:00:53.614417 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.614382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" event={"ID":"240acc86-45eb-4e34-aa6d-1884e9699cce","Type":"ContainerStarted","Data":"07775d83ce874a1a3783f61c7ec28494f3a565ecf5c0ef1324822acd63c7100a"} Sep 11 12:00:53.614603 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:53.614443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:54.520623 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:54.520573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6h82f_33290017-2b75-42ad-be09-29ee5b7e746e/dns/0.log" Sep 11 12:00:54.544219 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:54.544176 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6h82f_33290017-2b75-42ad-be09-29ee5b7e746e/kube-rbac-proxy/0.log" Sep 11 12:00:54.647916 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:54.647888 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fzqcl_239283ff-d4c3-469a-b910-44d112a3c32a/dns-node-resolver/0.log" Sep 11 12:00:55.108900 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:55.108868 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4s48h_3499a298-490e-489b-a5fb-307cd8fcf6e9/node-ca/0.log" Sep 11 12:00:56.428366 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:56.428339 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-shx9t_bbc8c5a1-ddea-4819-909d-23aae4d3e4f7/serve-healthcheck-canary/0.log" Sep 11 12:00:57.023821 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:57.023795 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrnsq_18b1ca60-6452-4e15-a673-3be9f7132d64/kube-rbac-proxy/0.log" Sep 11 12:00:57.060546 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:57.060516 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrnsq_18b1ca60-6452-4e15-a673-3be9f7132d64/exporter/0.log" Sep 11 12:00:57.087533 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:57.087512 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrnsq_18b1ca60-6452-4e15-a673-3be9f7132d64/extractor/0.log" Sep 11 12:00:59.630536 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:59.630510 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" Sep 11 12:00:59.662265 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:00:59.662212 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-54pxb/perf-node-gather-daemonset-9wwln" podStartSLOduration=7.662196606 podStartE2EDuration="7.662196606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:00:53.662733571 +0000 UTC m=+1830.198640559" watchObservedRunningTime="2026-09-11 12:00:59.662196606 +0000 UTC m=+1836.198103616" Sep 11 12:01:03.157847 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.157820 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7j8x7_469543e0-034f-49a3-ab0f-9f2ddeaf8fb8/kube-multus/0.log" Sep 11 12:01:03.392389 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.392363 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kkp9_e09db19d-ecec-41b5-85e3-164979161e08/kube-multus-additional-cni-plugins/0.log" Sep 11 12:01:03.415052 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.414982 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kkp9_e09db19d-ecec-41b5-85e3-164979161e08/egress-router-binary-copy/0.log" Sep 11 12:01:03.440072 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.440048 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kkp9_e09db19d-ecec-41b5-85e3-164979161e08/cni-plugins/0.log" Sep 11 12:01:03.467065 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.467041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kkp9_e09db19d-ecec-41b5-85e3-164979161e08/bond-cni-plugin/0.log" Sep 11 12:01:03.490251 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.490215 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kkp9_e09db19d-ecec-41b5-85e3-164979161e08/routeoverride-cni/0.log" Sep 11 12:01:03.527078 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.527034 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kkp9_e09db19d-ecec-41b5-85e3-164979161e08/whereabouts-cni-bincopy/0.log" Sep 11 12:01:03.582154 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.582130 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8kkp9_e09db19d-ecec-41b5-85e3-164979161e08/whereabouts-cni/0.log" Sep 11 12:01:03.967006 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:03.966966 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-59gwj_7eafd60c-bb2d-4473-ab8a-76180f836548/network-metrics-daemon/0.log" Sep 11 12:01:04.001636 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:04.001561 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-59gwj_7eafd60c-bb2d-4473-ab8a-76180f836548/kube-rbac-proxy/0.log" Sep 11 12:01:05.195823 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.195785 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/ovn-controller/0.log" Sep 11 12:01:05.217083 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.217054 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/ovn-acl-logging/0.log" Sep 11 12:01:05.243646 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.243618 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/kube-rbac-proxy-node/0.log" Sep 11 12:01:05.270458 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.270438 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/kube-rbac-proxy-ovn-metrics/0.log" Sep 11 12:01:05.295753 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.295711 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/northd/0.log" Sep 11 12:01:05.323948 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.323925 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/nbdb/0.log" Sep 11 12:01:05.355832 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.355815 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/sbdb/0.log" Sep 11 12:01:05.416282 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:05.416258 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zcd47_77a32e00-3aac-4894-97c2-42327fc00c08/ovnkube-controller/0.log" Sep 11 12:01:06.712124 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:06.712092 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-prvf7_ba4cf6d4-6127-4838-8e66-f71984a95365/network-check-target-container/0.log" Sep 11 12:01:07.569865 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:07.569838 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lp96t_7d82b26a-e9ab-4593-9179-fa7a73dd9cf6/iptables-alerter/0.log" Sep 11 12:01:08.239890 ip-10-0-142-17 kubenswrapper[2579]: I0911 12:01:08.239852 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-k5mxn_5b67833d-ab3e-4a80-b687-3ab730fd8a3e/tuned/0.log"