May 28 15:10:27.369969 ip-10-0-130-98 systemd[1]: Starting Kubernetes Kubelet... May 28 15:10:27.847296 ip-10-0-130-98 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 28 15:10:27.847296 ip-10-0-130-98 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 28 15:10:27.847296 ip-10-0-130-98 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 28 15:10:27.847296 ip-10-0-130-98 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 28 15:10:27.847296 ip-10-0-130-98 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 28 15:10:27.850390 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.850306 2569 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 28 15:10:27.853022 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853007 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 28 15:10:27.853022 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853022 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853026 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853030 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853034 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853037 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853041 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853046 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853049 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853052 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853054 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853057 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853060 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853062 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853065 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853067 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853070 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853073 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853075 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853078 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 28 15:10:27.853086 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853081 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853084 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853087 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853096 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853099 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853102 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853105 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853107 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853110 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853113 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853115 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853118 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853121 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853123 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853126 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853128 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853131 2569 feature_gate.go:328] unrecognized feature gate: Example May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853134 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853136 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853139 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 28 15:10:27.853524 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853141 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853143 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853146 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853148 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853151 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853153 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853156 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853159 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853162 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853165 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853168 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853170 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853172 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853175 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853178 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853181 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853184 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853187 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853190 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853194 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 28 15:10:27.854027 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853198 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853202 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853205 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853209 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853213 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853216 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853219 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853221 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853224 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853227 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853230 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853233 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853235 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853238 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853241 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853243 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853245 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853249 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853252 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853255 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 28 15:10:27.854514 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853258 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853261 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853263 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853266 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853268 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853271 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853673 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853678 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853680 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853683 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853686 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853688 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853691 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853694 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853696 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853700 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853703 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853707 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853710 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853713 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 28 15:10:27.855023 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853715 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853718 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853721 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853724 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853726 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853729 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853732 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853734 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853736 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853739 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853742 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853745 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853747 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853750 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853752 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853755 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853757 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853760 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853763 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853766 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 28 15:10:27.855502 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853768 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853771 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853773 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853776 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853778 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853781 2569 feature_gate.go:328] unrecognized feature gate: Example May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853785 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853789 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853791 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853794 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853796 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853799 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853802 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853804 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853806 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853809 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853812 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853815 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853817 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853820 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 28 15:10:27.856033 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853822 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853824 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853827 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853829 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853832 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853834 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853837 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853839 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853842 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853845 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853848 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853851 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853853 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853856 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853858 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853861 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853863 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853866 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853868 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 28 15:10:27.856531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853871 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853874 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853876 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853878 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853881 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853884 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853886 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853888 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853891 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853894 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853897 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853901 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.853903 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854548 2569 flags.go:64] FLAG: --address="0.0.0.0" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854562 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854568 2569 flags.go:64] FLAG: --anonymous-auth="true" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854573 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854577 2569 flags.go:64] FLAG: --authentication-token-webhook="false" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854580 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854585 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854589 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 28 15:10:27.857017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854592 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854596 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854600 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854603 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854606 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854610 2569 flags.go:64] FLAG: --cgroup-root="" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854613 2569 flags.go:64] FLAG: --cgroups-per-qos="true" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854628 2569 flags.go:64] FLAG: --client-ca-file="" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854631 2569 flags.go:64] FLAG: --cloud-config="" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854634 2569 flags.go:64] FLAG: --cloud-provider="external" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854638 2569 flags.go:64] FLAG: --cluster-dns="[]" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854643 2569 flags.go:64] FLAG: --cluster-domain="" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854646 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854650 2569 flags.go:64] FLAG: --config-dir="" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854653 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854657 2569 flags.go:64] FLAG: --container-log-max-files="5" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854660 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854664 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854667 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854670 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854674 2569 flags.go:64] FLAG: --contention-profiling="false" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854678 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854681 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854684 2569 flags.go:64] FLAG: --cpu-manager-policy="none" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854687 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" May 28 15:10:27.857546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854691 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854694 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854697 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854700 2569 flags.go:64] FLAG: --enable-load-reader="false" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854703 2569 flags.go:64] FLAG: --enable-server="true" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854706 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854711 2569 flags.go:64] FLAG: --event-burst="100" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854714 2569 flags.go:64] FLAG: --event-qps="50" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854717 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854720 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854724 2569 flags.go:64] FLAG: --eviction-hard="" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854727 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854730 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854733 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854736 2569 flags.go:64] FLAG: --eviction-soft="" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854739 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854742 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854745 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854749 2569 flags.go:64] FLAG: --experimental-mounter-path="" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854751 2569 flags.go:64] FLAG: --fail-cgroupv1="false" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854755 2569 flags.go:64] FLAG: --fail-swap-on="true" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854758 2569 flags.go:64] FLAG: --feature-gates="" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854762 2569 flags.go:64] FLAG: --file-check-frequency="20s" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854765 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854768 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 28 15:10:27.858169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854771 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854774 2569 flags.go:64] FLAG: --healthz-port="10248" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854777 2569 flags.go:64] FLAG: --help="false" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854780 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-130-98.ec2.internal" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854784 2569 flags.go:64] FLAG: --housekeeping-interval="10s" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854787 2569 flags.go:64] FLAG: --http-check-frequency="20s" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854790 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854793 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854797 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854799 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854802 2569 flags.go:64] FLAG: --image-service-endpoint="" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854805 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854808 2569 flags.go:64] FLAG: --kube-api-burst="100" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854811 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854814 2569 flags.go:64] FLAG: --kube-api-qps="50" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854820 2569 flags.go:64] FLAG: --kube-reserved="" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854823 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854826 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854829 2569 flags.go:64] FLAG: --kubelet-cgroups="" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854832 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854835 2569 flags.go:64] FLAG: --lock-file="" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854838 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854841 2569 flags.go:64] FLAG: --log-flush-frequency="5s" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854844 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 28 15:10:27.858779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854850 2569 flags.go:64] FLAG: --log-json-split-stream="false" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854853 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854856 2569 flags.go:64] FLAG: --log-text-split-stream="false" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854859 2569 flags.go:64] FLAG: --logging-format="text" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854862 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854865 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854868 2569 flags.go:64] FLAG: --manifest-url="" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854871 2569 flags.go:64] FLAG: --manifest-url-header="" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854875 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854878 2569 flags.go:64] FLAG: --max-open-files="1000000" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854882 2569 flags.go:64] FLAG: --max-pods="110" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854885 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854888 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854892 2569 flags.go:64] FLAG: --memory-manager-policy="None" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854895 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854899 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854903 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854908 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854920 2569 flags.go:64] FLAG: --node-status-max-images="50" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854925 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854930 2569 flags.go:64] FLAG: --oom-score-adj="-999" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854934 2569 flags.go:64] FLAG: --pod-cidr="" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854936 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 28 15:10:27.859420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854944 2569 flags.go:64] FLAG: --pod-manifest-path="" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854948 2569 flags.go:64] FLAG: --pod-max-pids="-1" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854951 2569 flags.go:64] FLAG: --pods-per-core="0" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854954 2569 flags.go:64] FLAG: --port="10250" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854957 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854960 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ea8ef2400a38f218" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854963 2569 flags.go:64] FLAG: --qos-reserved="" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854966 2569 flags.go:64] FLAG: --read-only-port="10255" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854969 2569 flags.go:64] FLAG: --register-node="true" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854972 2569 flags.go:64] FLAG: --register-schedulable="true" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854975 2569 flags.go:64] FLAG: --register-with-taints="" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854979 2569 flags.go:64] FLAG: --registry-burst="10" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854982 2569 flags.go:64] FLAG: --registry-qps="5" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854984 2569 flags.go:64] FLAG: --reserved-cpus="" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854987 2569 flags.go:64] FLAG: --reserved-memory="" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854991 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854994 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854997 2569 flags.go:64] FLAG: --rotate-certificates="false" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.854999 2569 flags.go:64] FLAG: --rotate-server-certificates="false" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855002 2569 flags.go:64] FLAG: --runonce="false" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855005 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855008 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855011 2569 flags.go:64] FLAG: --seccomp-default="false" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855015 2569 flags.go:64] FLAG: --serialize-image-pulls="true" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855018 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855021 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 28 15:10:27.860011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855024 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855027 2569 flags.go:64] FLAG: --storage-driver-password="root" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855030 2569 flags.go:64] FLAG: --storage-driver-secure="false" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855033 2569 flags.go:64] FLAG: --storage-driver-table="stats" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855036 2569 flags.go:64] FLAG: --storage-driver-user="root" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855039 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855043 2569 flags.go:64] FLAG: --sync-frequency="1m0s" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855046 2569 flags.go:64] FLAG: --system-cgroups="" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855049 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855054 2569 flags.go:64] FLAG: --system-reserved-cgroup="" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855057 2569 flags.go:64] FLAG: --tls-cert-file="" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855060 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855066 2569 flags.go:64] FLAG: --tls-min-version="" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855068 2569 flags.go:64] FLAG: --tls-private-key-file="" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855071 2569 flags.go:64] FLAG: --topology-manager-policy="none" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855074 2569 flags.go:64] FLAG: --topology-manager-policy-options="" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855077 2569 flags.go:64] FLAG: --topology-manager-scope="container" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855082 2569 flags.go:64] FLAG: --v="2" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855086 2569 flags.go:64] FLAG: --version="false" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855094 2569 flags.go:64] FLAG: --vmodule="" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855098 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.855102 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855196 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855199 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 28 15:10:27.860698 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855202 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855205 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855208 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855211 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855214 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855216 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855220 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855222 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855225 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855227 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855230 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855233 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855235 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855238 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855241 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855243 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855246 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855249 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855251 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855254 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 28 15:10:27.861279 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855257 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855259 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855262 2569 feature_gate.go:328] unrecognized feature gate: Example May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855264 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855267 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855271 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855274 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855276 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855279 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855282 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855284 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855287 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855290 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855292 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855295 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855298 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855300 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855303 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855306 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855309 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 28 15:10:27.861790 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855312 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855314 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855317 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855320 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855322 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855325 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855328 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855331 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855334 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855336 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855339 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855342 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855345 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855347 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855350 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855352 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855355 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855358 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855361 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 28 15:10:27.862305 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855363 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855366 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855369 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855371 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855374 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855376 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855379 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855382 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855384 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855387 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855389 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855392 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855395 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855398 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855402 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855406 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855409 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855412 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855414 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 28 15:10:27.862800 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855418 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855422 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855425 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855428 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855431 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.855433 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.856089 2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.863227 2569 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 28 15:10:27.863271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.863242 2569 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863288 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863292 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863296 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863299 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863302 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863306 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863310 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863313 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863316 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863318 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863321 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863324 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863326 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863329 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863332 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863334 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863337 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863340 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 28 15:10:27.863479 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863343 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863345 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863348 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863350 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863353 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863356 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863358 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863361 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863364 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863366 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863369 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863371 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863374 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863377 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863379 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863383 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863386 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863389 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863391 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863394 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 28 15:10:27.864009 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863396 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863399 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863402 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863404 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863407 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863411 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863416 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863419 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863421 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863424 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863427 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863430 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863434 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863437 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863439 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863442 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863445 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863447 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863450 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 28 15:10:27.864588 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863453 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863455 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863458 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863460 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863463 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863466 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863468 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863471 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863474 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863476 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863479 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863482 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863485 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863487 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863490 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863493 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863495 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863497 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863500 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 28 15:10:27.865143 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863502 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863505 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863508 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863510 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863513 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863515 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863518 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863521 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863524 2569 feature_gate.go:328] unrecognized feature gate: Example May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863526 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.863532 2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863648 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863653 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863656 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863661 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 28 15:10:27.865612 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863664 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863667 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863670 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863673 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863676 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863679 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863681 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863684 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863687 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863690 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863692 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863695 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863697 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863700 2569 feature_gate.go:328] unrecognized feature gate: Example May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863703 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863705 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863708 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863710 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863713 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863715 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 28 15:10:27.866000 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863717 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863720 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863723 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863726 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863728 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863731 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863734 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863736 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863739 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863741 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863743 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863746 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863748 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863751 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863754 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863757 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863759 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863762 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863764 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863767 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 28 15:10:27.866492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863769 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863772 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863774 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863777 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863780 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863782 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863785 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863787 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863789 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863792 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863794 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863797 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863799 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863802 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863804 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863807 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863809 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863812 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863815 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 28 15:10:27.867005 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863818 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863821 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863823 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863826 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863828 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863831 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863834 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863836 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863839 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863841 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863844 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863846 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863849 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863851 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863853 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863856 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863859 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863862 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863864 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863867 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 28 15:10:27.867549 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863869 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 28 15:10:27.868061 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863872 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 28 15:10:27.868061 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:27.863875 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 28 15:10:27.868061 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.863880 2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 28 15:10:27.868061 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.864008 2569 server.go:962] "Client rotation is on, will bootstrap in background" May 28 15:10:27.868061 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.866643 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 28 15:10:27.868061 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.867568 2569 server.go:1019] "Starting client certificate rotation" May 28 15:10:27.868061 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.867651 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 28 15:10:27.868526 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.868514 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 28 15:10:27.894389 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.894366 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 28 15:10:27.896503 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.896472 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 28 15:10:27.914261 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.914102 2569 log.go:25] "Validated CRI v1 runtime API" May 28 15:10:27.919213 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.919193 2569 log.go:25] "Validated CRI v1 image API" May 28 15:10:27.920478 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.920458 2569 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 28 15:10:27.922741 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.922714 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bf0e164d-564d-4188-89ed-c37fa5cf456f:/dev/nvme0n1p3 dcda8a2d-9398-4bde-b0b2-a64219b430bd:/dev/nvme0n1p4] May 28 15:10:27.922800 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.922742 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 28 15:10:27.923280 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.923261 2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 28 15:10:27.927250 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.927142 2569 manager.go:217] Machine: {Timestamp:2026-05-28 15:10:27.926652866 +0000 UTC m=+0.429407017 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100109 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27970163fa650b967b845b5002111e SystemUUID:ec279701-63fa-650b-967b-845b5002111e BootID:e0bf8597-873c-49cb-8159-f5d2a12bf2b6 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:49:fd:2d:cf:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:49:fd:2d:cf:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:9b:3a:ea:bb:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 28 15:10:27.927880 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.927869 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 28 15:10:27.927984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.927970 2569 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 28 15:10:27.932231 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.932203 2569 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 28 15:10:27.932372 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.932234 2569 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 28 15:10:27.932419 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.932382 2569 topology_manager.go:138] "Creating topology manager with none policy" May 28 15:10:27.932419 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.932391 2569 container_manager_linux.go:306] "Creating device plugin manager" May 28 15:10:27.932419 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.932401 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 28 15:10:27.933019 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.933008 2569 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 28 15:10:27.934203 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.934191 2569 state_mem.go:36] "Initialized new in-memory state store" May 28 15:10:27.934314 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.934305 2569 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 28 15:10:27.937311 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.937299 2569 kubelet.go:491] "Attempting to sync node with API server" May 28 15:10:27.937355 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.937316 2569 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 28 15:10:27.937355 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.937330 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 28 15:10:27.937355 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.937343 2569 kubelet.go:397] "Adding apiserver pod source" May 28 15:10:27.937355 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.937354 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 28 15:10:27.938539 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.938527 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 28 15:10:27.938592 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.938548 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 28 15:10:27.941360 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.941341 2569 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 28 15:10:27.942982 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.942970 2569 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 28 15:10:27.943968 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.943950 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 28 15:10:27.943968 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.943970 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 28 15:10:27.944121 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.943977 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 28 15:10:27.944121 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.943983 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 28 15:10:27.944121 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.943992 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 28 15:10:27.945039 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.944000 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 28 15:10:27.945118 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.945051 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 28 15:10:27.945118 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.945066 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 28 15:10:27.945118 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.945080 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 28 15:10:27.945118 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.945097 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 28 15:10:27.945118 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.945112 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 28 15:10:27.945331 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.945127 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 28 15:10:27.946831 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.946812 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 28 15:10:27.946929 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.946844 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 28 15:10:27.950290 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.950273 2569 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 28 15:10:27.950363 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.950320 2569 server.go:1295] "Started kubelet" May 28 15:10:27.950423 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.950381 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 28 15:10:27.950475 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.950407 2569 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 28 15:10:27.950530 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.950486 2569 server_v1.go:47] "podresources" method="list" useActivePods=true May 28 15:10:27.950575 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.950565 2569 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 28 15:10:27.950693 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.950674 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 28 15:10:27.950743 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.950731 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 28 15:10:27.951211 ip-10-0-130-98 systemd[1]: Started Kubernetes Kubelet. May 28 15:10:27.951800 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.951754 2569 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 28 15:10:27.953032 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.953016 2569 server.go:317] "Adding debug handlers to kubelet server" May 28 15:10:27.957889 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.957867 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 28 15:10:27.958467 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.958447 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 28 15:10:27.958633 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.958598 2569 volume_manager.go:295] "The desired_state_of_world populator starts" May 28 15:10:27.958633 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.958634 2569 volume_manager.go:297] "Starting Kubelet Volume Manager" May 28 15:10:27.958772 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.957843 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-98.ec2.internal.18b3c3506de96249 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-98.ec2.internal,UID:ip-10-0-130-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-98.ec2.internal,},FirstTimestamp:2026-05-28 15:10:27.950289481 +0000 UTC m=+0.453043640,LastTimestamp:2026-05-28 15:10:27.950289481 +0000 UTC m=+0.453043640,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-98.ec2.internal,}" May 28 15:10:27.958772 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.958746 2569 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 28 15:10:27.958892 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.958794 2569 reconstruct.go:97] "Volume reconstruction finished" May 28 15:10:27.958892 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.958801 2569 reconciler.go:26] "Reconciler: start to sync state" May 28 15:10:27.959224 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.959206 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:27.959549 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.959534 2569 factory.go:55] Registering systemd factory May 28 15:10:27.959664 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.959555 2569 factory.go:223] Registration of the systemd container factory successfully May 28 15:10:27.959864 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.959849 2569 factory.go:153] Registering CRI-O factory May 28 15:10:27.959864 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.959864 2569 factory.go:223] Registration of the crio container factory successfully May 28 15:10:27.959988 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.959933 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 28 15:10:27.959988 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.959954 2569 factory.go:103] Registering Raw factory May 28 15:10:27.959988 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.959974 2569 manager.go:1196] Started watching for new ooms in manager May 28 15:10:27.960269 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.960259 2569 manager.go:319] Starting recovery of all containers May 28 15:10:27.960816 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.960798 2569 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 28 15:10:27.964002 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.963089 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" May 28 15:10:27.964002 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.963211 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" May 28 15:10:27.964597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.964572 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8nzt9" May 28 15:10:27.969005 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.968984 2569 manager.go:324] Recovery completed May 28 15:10:27.971334 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.971283 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8nzt9" May 28 15:10:27.976134 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.976122 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 28 15:10:27.977944 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.977928 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientMemory" May 28 15:10:27.978023 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.977955 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasNoDiskPressure" May 28 15:10:27.978023 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.977966 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientPID" May 28 15:10:27.978490 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.978477 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" May 28 15:10:27.978490 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.978489 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 28 15:10:27.978567 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.978532 2569 state_mem.go:36] "Initialized new in-memory state store" May 28 15:10:27.980172 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:27.980105 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-98.ec2.internal.18b3c3506f8f58f6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-98.ec2.internal,UID:ip-10-0-130-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-98.ec2.internal,},FirstTimestamp:2026-05-28 15:10:27.977943286 +0000 UTC m=+0.480697438,LastTimestamp:2026-05-28 15:10:27.977943286 +0000 UTC m=+0.480697438,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-98.ec2.internal,}" May 28 15:10:27.980761 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.980748 2569 policy_none.go:49] "None policy: Start" May 28 15:10:27.980812 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.980766 2569 memory_manager.go:186] "Starting memorymanager" policy="None" May 28 15:10:27.980812 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:27.980776 2569 state_mem.go:35] "Initializing new in-memory state store" May 28 15:10:28.028440 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.028416 2569 manager.go:341] "Starting Device Plugin manager" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.028456 2569 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.028470 2569 server.go:85] "Starting device plugin registration server" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.028768 2569 eviction_manager.go:189] "Eviction manager: starting control loop" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.028781 2569 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.028872 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.029018 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.029027 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.029924 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 28 15:10:28.038668 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.029966 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.067159 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.067114 2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 28 15:10:28.068284 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.068262 2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 28 15:10:28.068366 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.068290 2569 status_manager.go:230] "Starting to sync pod status with apiserver" May 28 15:10:28.068366 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.068309 2569 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 28 15:10:28.068366 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.068315 2569 kubelet.go:2451] "Starting kubelet main sync loop" May 28 15:10:28.068366 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.068352 2569 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 28 15:10:28.070929 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.070909 2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 28 15:10:28.128985 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.128909 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 28 15:10:28.130466 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.130451 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientMemory" May 28 15:10:28.130534 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.130482 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasNoDiskPressure" May 28 15:10:28.130534 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.130492 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientPID" May 28 15:10:28.130534 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.130515 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-98.ec2.internal" May 28 15:10:28.136905 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.136881 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-98.ec2.internal" May 28 15:10:28.136905 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.136906 2569 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-98.ec2.internal\": node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.154940 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.154912 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.168428 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.168398 2569 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal"] May 28 15:10:28.168565 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.168479 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 28 15:10:28.170356 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.170338 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientMemory" May 28 15:10:28.170463 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.170373 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasNoDiskPressure" May 28 15:10:28.170463 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.170416 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientPID" May 28 15:10:28.171833 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.171817 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 28 15:10:28.172017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172002 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.172094 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172032 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 28 15:10:28.172526 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172508 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientMemory" May 28 15:10:28.172606 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172532 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasNoDiskPressure" May 28 15:10:28.172606 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172530 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientMemory" May 28 15:10:28.172606 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172562 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasNoDiskPressure" May 28 15:10:28.172606 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172571 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientPID" May 28 15:10:28.172766 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.172544 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientPID" May 28 15:10:28.173891 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.173874 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" May 28 15:10:28.173952 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.173909 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 28 15:10:28.174515 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.174499 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientMemory" May 28 15:10:28.174592 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.174524 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasNoDiskPressure" May 28 15:10:28.174592 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.174533 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeHasSufficientPID" May 28 15:10:28.204454 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.204421 2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-98.ec2.internal\" not found" node="ip-10-0-130-98.ec2.internal" May 28 15:10:28.208821 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.208802 2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-98.ec2.internal\" not found" node="ip-10-0-130-98.ec2.internal" May 28 15:10:28.255921 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.255885 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.260199 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.260183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a61332edee42088905ed1b7a0d6356e8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-98.ec2.internal\" (UID: \"a61332edee42088905ed1b7a0d6356e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" May 28 15:10:28.260284 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.260208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a47726c6874c43b16c1bd225ab259c42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal\" (UID: \"a47726c6874c43b16c1bd225ab259c42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.260284 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.260231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a47726c6874c43b16c1bd225ab259c42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal\" (UID: \"a47726c6874c43b16c1bd225ab259c42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.356369 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.356334 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.360677 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.360643 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a61332edee42088905ed1b7a0d6356e8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-98.ec2.internal\" (UID: \"a61332edee42088905ed1b7a0d6356e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" May 28 15:10:28.360677 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.360680 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a47726c6874c43b16c1bd225ab259c42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal\" (UID: \"a47726c6874c43b16c1bd225ab259c42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.360847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.360698 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a47726c6874c43b16c1bd225ab259c42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal\" (UID: \"a47726c6874c43b16c1bd225ab259c42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.360847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.360744 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a47726c6874c43b16c1bd225ab259c42-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal\" (UID: \"a47726c6874c43b16c1bd225ab259c42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.360847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.360749 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a61332edee42088905ed1b7a0d6356e8-config\") pod \"kube-apiserver-proxy-ip-10-0-130-98.ec2.internal\" (UID: \"a61332edee42088905ed1b7a0d6356e8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" May 28 15:10:28.360847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.360753 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a47726c6874c43b16c1bd225ab259c42-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal\" (UID: \"a47726c6874c43b16c1bd225ab259c42\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.457148 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.457060 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.507560 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.507523 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:28.511345 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.511321 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" May 28 15:10:28.558113 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.558068 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.658509 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.658458 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.758952 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.758872 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.819711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.819681 2569 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 28 15:10:28.859783 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.859752 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.867003 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.866977 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 28 15:10:28.867147 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.867124 2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 28 15:10:28.867188 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.867150 2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 28 15:10:28.958943 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.958911 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 28 15:10:28.959956 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:28.959941 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:28.968023 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.968002 2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 28 15:10:28.974753 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.974722 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-27 15:05:27 +0000 UTC" deadline="2027-12-21 01:28:49.09473628 +0000 UTC" May 28 15:10:28.974753 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.974752 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13714h18m20.119987308s" May 28 15:10:28.986664 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:28.986614 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61332edee42088905ed1b7a0d6356e8.slice/crio-c18562f8597b9f372539333462d63ff59ec8c7bca34e5d59cdb11cf745d8dc29 WatchSource:0}: Error finding container c18562f8597b9f372539333462d63ff59ec8c7bca34e5d59cdb11cf745d8dc29: Status 404 returned error can't find the container with id c18562f8597b9f372539333462d63ff59ec8c7bca34e5d59cdb11cf745d8dc29 May 28 15:10:28.986991 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:28.986972 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47726c6874c43b16c1bd225ab259c42.slice/crio-c15afa3ac8076378d238e90651799543fbe41926bf74f5d0e1df068a2efc34e8 WatchSource:0}: Error finding container c15afa3ac8076378d238e90651799543fbe41926bf74f5d0e1df068a2efc34e8: Status 404 returned error can't find the container with id c15afa3ac8076378d238e90651799543fbe41926bf74f5d0e1df068a2efc34e8 May 28 15:10:28.987419 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.987395 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-spd8z" May 28 15:10:28.991219 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.991205 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 28 15:10:28.996765 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:28.996745 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-spd8z" May 28 15:10:29.060682 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:29.060578 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:29.071879 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.071826 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" event={"ID":"a61332edee42088905ed1b7a0d6356e8","Type":"ContainerStarted","Data":"c18562f8597b9f372539333462d63ff59ec8c7bca34e5d59cdb11cf745d8dc29"} May 28 15:10:29.072896 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.072875 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" event={"ID":"a47726c6874c43b16c1bd225ab259c42","Type":"ContainerStarted","Data":"c15afa3ac8076378d238e90651799543fbe41926bf74f5d0e1df068a2efc34e8"} May 28 15:10:29.128735 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.128709 2569 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 28 15:10:29.161026 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:29.160993 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-98.ec2.internal\" not found" May 28 15:10:29.185430 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.185402 2569 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 28 15:10:29.259608 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.259567 2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" May 28 15:10:29.270124 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.270101 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 28 15:10:29.270897 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.270885 2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" May 28 15:10:29.281652 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.281613 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 28 15:10:29.938594 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.938563 2569 apiserver.go:52] "Watching apiserver" May 28 15:10:29.946024 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.945998 2569 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 28 15:10:29.946398 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.946375 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-mndtq","openshift-network-operator/iptables-alerter-l6d25","kube-system/konnectivity-agent-bg5rp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal","openshift-multus/network-metrics-daemon-ckhtc","openshift-network-diagnostics/network-check-target-2vt4s","openshift-ovn-kubernetes/ovnkube-node-gx9nw","kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t","openshift-cluster-node-tuning-operator/tuned-zsdb5","openshift-dns/node-resolver-jlwmt","openshift-image-registry/node-ca-w7rh6","openshift-multus/multus-additional-cni-plugins-8vmcs"] May 28 15:10:29.949344 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.949312 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.950555 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.950521 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:29.951521 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.951499 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:29.951878 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.951855 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 28 15:10:29.951978 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.951860 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 28 15:10:29.951978 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.951936 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ps5p7\"" May 28 15:10:29.952737 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.952718 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:29.952859 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:29.952829 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:29.953281 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.953251 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 28 15:10:29.953684 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.953475 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 28 15:10:29.953684 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.953496 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 28 15:10:29.953862 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.953838 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:29.953926 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.953890 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2ft2j\"" May 28 15:10:29.953978 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:29.953925 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:29.954168 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.954149 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 28 15:10:29.954251 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.954200 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gqbv8\"" May 28 15:10:29.954569 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.954552 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 28 15:10:29.955238 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.955219 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.955324 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.955254 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.956745 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.956720 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mndtq" May 28 15:10:29.957958 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.957938 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:29.958479 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.958300 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 28 15:10:29.958479 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.958332 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 28 15:10:29.958640 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.958608 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 28 15:10:29.958691 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.958666 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 28 15:10:29.959604 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.959450 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:29.960231 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.960213 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 28 15:10:29.960640 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.960604 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 28 15:10:29.960781 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.960766 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 28 15:10:29.960850 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.960812 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xrm7w\"" May 28 15:10:29.960989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.960968 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 28 15:10:29.961098 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.961082 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 28 15:10:29.961184 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.961167 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 28 15:10:29.962031 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.962009 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 28 15:10:29.962459 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.962435 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnmdv\"" May 28 15:10:29.962753 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.962728 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 28 15:10:29.962998 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.962980 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:29.963279 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.963250 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 28 15:10:29.964184 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.964165 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tmn6w\"" May 28 15:10:29.964887 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.964863 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 28 15:10:29.964961 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.964937 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 28 15:10:29.965239 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.965219 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 28 15:10:29.965348 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.965330 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lhm88\"" May 28 15:10:29.965999 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.965980 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cbpfk\"" May 28 15:10:29.966186 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.966166 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 28 15:10:29.966259 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.966221 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 28 15:10:29.966308 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.966265 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 28 15:10:29.967376 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.967361 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 28 15:10:29.968530 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.968512 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9g5f5\"" May 28 15:10:29.969442 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-systemd\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.969527 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969458 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-kubelet\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.969527 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjh7p\" (UniqueName: \"kubernetes.io/projected/83f78757-e808-486a-9d6a-86aeccd930bb-kube-api-access-qjh7p\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:29.969527 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969510 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysconfig\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.969714 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysctl-conf\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.969714 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-systemd\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.969714 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969583 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.969714 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-etc-selinux\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.969714 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-var-lib-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.969714 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969675 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969721 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-system-cni-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-host\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969882 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-netns\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969909 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-cni-multus\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-device-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969965 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33dd655e-c457-41ba-b059-59c44998d358-cni-binary-copy\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.970001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.969994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjwf\" (UniqueName: \"kubernetes.io/projected/67fbf203-2dbe-401e-9d41-b9e841caa1e8-kube-api-access-pdjwf\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-kubernetes\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d09e9953-ec38-42d4-ac0b-353cfdd3683d-host\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-ovn\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83f78757-e808-486a-9d6a-86aeccd930bb-hosts-file\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-sys-fs\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-cni-bin\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-hostroot\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smn7q\" (UniqueName: \"kubernetes.io/projected/e5cb0634-a302-4dfd-a99d-5de8107704dc-kube-api-access-smn7q\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-etc-kubernetes\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970255 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-var-lib-kubelet\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-socket-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-registration-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.970386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-node-log\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-env-overrides\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovnkube-script-lib\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-multus-certs\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-lib-modules\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970540 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-run-netns\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovnkube-config\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgq64\" (UniqueName: \"kubernetes.io/projected/33dd655e-c457-41ba-b059-59c44998d358-kube-api-access-fgq64\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fb0dbb2-e7bc-4fdf-9582-b5709495834a-agent-certs\") pod \"konnectivity-agent-bg5rp\" (UID: \"8fb0dbb2-e7bc-4fdf-9582-b5709495834a\") " pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970757 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0dbb2-e7bc-4fdf-9582-b5709495834a-konnectivity-ca\") pod \"konnectivity-agent-bg5rp\" (UID: \"8fb0dbb2-e7bc-4fdf-9582-b5709495834a\") " pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-slash\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-etc-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970835 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33dd655e-c457-41ba-b059-59c44998d358-multus-daemon-config\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5352d147-37e5-4291-9fe9-66faf7e057c2-tmp\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.971092 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zvf\" (UniqueName: \"kubernetes.io/projected/5352d147-37e5-4291-9fe9-66faf7e057c2-kube-api-access-s7zvf\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdmk\" (UniqueName: \"kubernetes.io/projected/ed77076b-c4ca-4f11-934e-1f1ed7350613-kube-api-access-gxdmk\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.970977 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-log-socket\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-kubelet\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovn-node-metrics-cert\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971075 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-cnibin\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-cni-bin\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f78757-e808-486a-9d6a-86aeccd930bb-tmp-dir\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-run\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d09e9953-ec38-42d4-ac0b-353cfdd3683d-serviceca\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971194 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jkt\" (UniqueName: \"kubernetes.io/projected/d09e9953-ec38-42d4-ac0b-353cfdd3683d-kube-api-access-v2jkt\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971217 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-systemd-units\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-os-release\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-modprobe-d\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971280 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-sys\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971325 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5cb0634-a302-4dfd-a99d-5de8107704dc-host-slash\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:29.971911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6gg\" (UniqueName: \"kubernetes.io/projected/70961166-700d-40ad-867d-dcdf297e6704-kube-api-access-qb6gg\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-socket-dir-parent\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-k8s-cni-cncf-io\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-conf-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-cni-netd\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971477 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-cni-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971499 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysctl-d\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-tuned\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:29.972723 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.971546 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5cb0634-a302-4dfd-a99d-5de8107704dc-iptables-alerter-script\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:29.997480 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.997446 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 15:05:28 +0000 UTC" deadline="2028-01-07 11:38:49.389202627 +0000 UTC" May 28 15:10:29.997480 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:29.997477 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14132h28m19.391729145s" May 28 15:10:30.060083 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.060053 2569 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 28 15:10:30.071749 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071724 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-systemd\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071763 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-kubelet\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071787 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qjh7p\" (UniqueName: \"kubernetes.io/projected/83f78757-e808-486a-9d6a-86aeccd930bb-kube-api-access-qjh7p\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071810 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysconfig\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071831 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysctl-conf\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071837 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-systemd\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071855 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-kubelet\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071857 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-systemd\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071900 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysconfig\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.071916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071907 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-systemd\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071938 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071975 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-etc-selinux\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.071999 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-var-lib-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072004 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysctl-conf\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072025 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072030 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072045 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-etc-selinux\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072050 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-system-cni-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072069 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072074 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-var-lib-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072093 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-host\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072112 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-system-cni-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072142 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072153 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-host\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-os-release\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.072241 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:30.072342 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072240 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-netns\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.072307 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:30.572281282 +0000 UTC m=+3.075035421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072313 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-netns\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072352 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-cni-multus\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072397 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-cni-multus\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072418 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-device-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072476 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33dd655e-c457-41ba-b059-59c44998d358-cni-binary-copy\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nlt\" (UniqueName: \"kubernetes.io/projected/b848567f-fdd1-4b2c-82d2-f6692a9b4148-kube-api-access-55nlt\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072475 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-device-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072545 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjwf\" (UniqueName: \"kubernetes.io/projected/67fbf203-2dbe-401e-9d41-b9e841caa1e8-kube-api-access-pdjwf\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072566 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-kubernetes\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072582 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d09e9953-ec38-42d4-ac0b-353cfdd3683d-host\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072596 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-ovn\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072629 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83f78757-e808-486a-9d6a-86aeccd930bb-hosts-file\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072655 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-sys-fs\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072666 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-kubernetes\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d09e9953-ec38-42d4-ac0b-353cfdd3683d-host\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:30.073216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072691 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-cni-bin\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072718 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-hostroot\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072721 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83f78757-e808-486a-9d6a-86aeccd930bb-hosts-file\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072747 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-smn7q\" (UniqueName: \"kubernetes.io/projected/e5cb0634-a302-4dfd-a99d-5de8107704dc-kube-api-access-smn7q\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072765 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-run-ovn\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072812 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-cni-bin\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072809 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-etc-kubernetes\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072846 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-var-lib-kubelet\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072857 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-sys-fs\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072892 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-socket-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072921 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-registration-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072974 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-node-log\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.072993 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33dd655e-c457-41ba-b059-59c44998d358-cni-binary-copy\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073002 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073030 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-hostroot\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073028 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-env-overrides\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074021 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073065 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovnkube-script-lib\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073096 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073135 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-registration-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073136 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-node-log\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073156 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ed77076b-c4ca-4f11-934e-1f1ed7350613-socket-dir\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073161 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-multus-certs\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073139 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-var-lib-kubelet\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073197 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-lib-modules\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073211 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-etc-kubernetes\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073234 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-run-netns\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073238 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-multus-certs\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073319 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-lib-modules\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073387 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073418 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073482 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovnkube-config\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073509 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgq64\" (UniqueName: \"kubernetes.io/projected/33dd655e-c457-41ba-b059-59c44998d358-kube-api-access-fgq64\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073555 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fb0dbb2-e7bc-4fdf-9582-b5709495834a-agent-certs\") pod \"konnectivity-agent-bg5rp\" (UID: \"8fb0dbb2-e7bc-4fdf-9582-b5709495834a\") " pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:30.074828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-env-overrides\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073583 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0dbb2-e7bc-4fdf-9582-b5709495834a-konnectivity-ca\") pod \"konnectivity-agent-bg5rp\" (UID: \"8fb0dbb2-e7bc-4fdf-9582-b5709495834a\") " pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073637 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-run-netns\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073762 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-slash\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073795 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-etc-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073763 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovnkube-script-lib\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073892 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-etc-openvswitch\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073894 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-slash\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073933 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33dd655e-c457-41ba-b059-59c44998d358-multus-daemon-config\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073959 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5352d147-37e5-4291-9fe9-66faf7e057c2-tmp\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.073991 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zvf\" (UniqueName: \"kubernetes.io/projected/5352d147-37e5-4291-9fe9-66faf7e057c2-kube-api-access-s7zvf\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074018 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdmk\" (UniqueName: \"kubernetes.io/projected/ed77076b-c4ca-4f11-934e-1f1ed7350613-kube-api-access-gxdmk\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074029 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074067 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-log-socket\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074094 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-system-cni-dir\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.075681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074154 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cni-binary-copy\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074180 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-kubelet\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074204 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovn-node-metrics-cert\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074218 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovnkube-config\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074230 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-cnibin\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074256 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-cni-bin\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074280 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f78757-e808-486a-9d6a-86aeccd930bb-tmp-dir\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074292 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fb0dbb2-e7bc-4fdf-9582-b5709495834a-konnectivity-ca\") pod \"konnectivity-agent-bg5rp\" (UID: \"8fb0dbb2-e7bc-4fdf-9582-b5709495834a\") " pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074304 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-run\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074345 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d09e9953-ec38-42d4-ac0b-353cfdd3683d-serviceca\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-run\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074372 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jkt\" (UniqueName: \"kubernetes.io/projected/d09e9953-ec38-42d4-ac0b-353cfdd3683d-kube-api-access-v2jkt\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074406 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-kubelet\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074461 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33dd655e-c457-41ba-b059-59c44998d358-multus-daemon-config\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074464 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-systemd-units\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074407 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-systemd-units\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075018 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-os-release\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075053 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-modprobe-d\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.076502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075084 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-sys\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075109 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5cb0634-a302-4dfd-a99d-5de8107704dc-host-slash\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075136 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6gg\" (UniqueName: \"kubernetes.io/projected/70961166-700d-40ad-867d-dcdf297e6704-kube-api-access-qb6gg\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075159 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-socket-dir-parent\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075180 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-k8s-cni-cncf-io\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075202 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-conf-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075217 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-modprobe-d\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074776 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-var-lib-cni-bin\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075231 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-cni-netd\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075241 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-os-release\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074765 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f78757-e808-486a-9d6a-86aeccd930bb-tmp-dir\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075268 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-host-cni-netd\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075267 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-cni-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075316 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysctl-d\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074972 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-cnibin\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075337 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5cb0634-a302-4dfd-a99d-5de8107704dc-host-slash\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075343 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-tuned\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075318 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-cni-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.077285 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.074981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/67fbf203-2dbe-401e-9d41-b9e841caa1e8-log-socket\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075338 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d09e9953-ec38-42d4-ac0b-353cfdd3683d-serviceca\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075397 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-conf-dir\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075402 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-multus-socket-dir-parent\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075444 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5cb0634-a302-4dfd-a99d-5de8107704dc-iptables-alerter-script\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075464 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-sys\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cnibin\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075549 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-sysctl-d\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.075564 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33dd655e-c457-41ba-b059-59c44998d358-host-run-k8s-cni-cncf-io\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.078049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.076008 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5cb0634-a302-4dfd-a99d-5de8107704dc-iptables-alerter-script\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:30.078530 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.078417 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5352d147-37e5-4291-9fe9-66faf7e057c2-tmp\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.079027 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.078911 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fb0dbb2-e7bc-4fdf-9582-b5709495834a-agent-certs\") pod \"konnectivity-agent-bg5rp\" (UID: \"8fb0dbb2-e7bc-4fdf-9582-b5709495834a\") " pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:30.080176 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.079336 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/67fbf203-2dbe-401e-9d41-b9e841caa1e8-ovn-node-metrics-cert\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.080793 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.080723 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5352d147-37e5-4291-9fe9-66faf7e057c2-etc-tuned\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.080876 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.080847 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 28 15:10:30.080876 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.080871 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 28 15:10:30.080967 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.080885 2569 projected.go:194] Error preparing data for projected volume kube-api-access-v4qkr for pod openshift-network-diagnostics/network-check-target-2vt4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:30.080967 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.080950 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr podName:e018896b-bc34-4bda-b761-752548a2cde1 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:30.580933108 +0000 UTC m=+3.083687268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v4qkr" (UniqueName: "kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr") pod "network-check-target-2vt4s" (UID: "e018896b-bc34-4bda-b761-752548a2cde1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:30.083211 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.083191 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smn7q\" (UniqueName: \"kubernetes.io/projected/e5cb0634-a302-4dfd-a99d-5de8107704dc-kube-api-access-smn7q\") pod \"iptables-alerter-l6d25\" (UID: \"e5cb0634-a302-4dfd-a99d-5de8107704dc\") " pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:30.086400 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.086376 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zvf\" (UniqueName: \"kubernetes.io/projected/5352d147-37e5-4291-9fe9-66faf7e057c2-kube-api-access-s7zvf\") pod \"tuned-zsdb5\" (UID: \"5352d147-37e5-4291-9fe9-66faf7e057c2\") " pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.086400 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.086392 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjh7p\" (UniqueName: \"kubernetes.io/projected/83f78757-e808-486a-9d6a-86aeccd930bb-kube-api-access-qjh7p\") pod \"node-resolver-jlwmt\" (UID: \"83f78757-e808-486a-9d6a-86aeccd930bb\") " pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:30.086547 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.086382 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjwf\" (UniqueName: \"kubernetes.io/projected/67fbf203-2dbe-401e-9d41-b9e841caa1e8-kube-api-access-pdjwf\") pod \"ovnkube-node-gx9nw\" (UID: \"67fbf203-2dbe-401e-9d41-b9e841caa1e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.087030 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.086991 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdmk\" (UniqueName: \"kubernetes.io/projected/ed77076b-c4ca-4f11-934e-1f1ed7350613-kube-api-access-gxdmk\") pod \"aws-ebs-csi-driver-node-sq46t\" (UID: \"ed77076b-c4ca-4f11-934e-1f1ed7350613\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.087199 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.087110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgq64\" (UniqueName: \"kubernetes.io/projected/33dd655e-c457-41ba-b059-59c44998d358-kube-api-access-fgq64\") pod \"multus-mndtq\" (UID: \"33dd655e-c457-41ba-b059-59c44998d358\") " pod="openshift-multus/multus-mndtq" May 28 15:10:30.088558 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.088520 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6gg\" (UniqueName: \"kubernetes.io/projected/70961166-700d-40ad-867d-dcdf297e6704-kube-api-access-qb6gg\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:30.089111 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.089090 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jkt\" (UniqueName: \"kubernetes.io/projected/d09e9953-ec38-42d4-ac0b-353cfdd3683d-kube-api-access-v2jkt\") pod \"node-ca-w7rh6\" (UID: \"d09e9953-ec38-42d4-ac0b-353cfdd3683d\") " pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:30.175907 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.175872 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55nlt\" (UniqueName: \"kubernetes.io/projected/b848567f-fdd1-4b2c-82d2-f6692a9b4148-kube-api-access-55nlt\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176080 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.175925 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176080 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.175957 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176080 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.175997 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-system-cni-dir\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176080 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176021 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cni-binary-copy\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176080 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176059 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cnibin\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176325 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176082 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-system-cni-dir\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176325 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176097 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-os-release\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176325 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176125 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176325 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176136 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176325 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176184 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cnibin\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176325 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176232 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b848567f-fdd1-4b2c-82d2-f6692a9b4148-os-release\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176611 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176499 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176697 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176666 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-cni-binary-copy\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.176697 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.176673 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b848567f-fdd1-4b2c-82d2-f6692a9b4148-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.183963 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.183940 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nlt\" (UniqueName: \"kubernetes.io/projected/b848567f-fdd1-4b2c-82d2-f6692a9b4148-kube-api-access-55nlt\") pod \"multus-additional-cni-plugins-8vmcs\" (UID: \"b848567f-fdd1-4b2c-82d2-f6692a9b4148\") " pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.265033 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.264962 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" May 28 15:10:30.275727 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.275695 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l6d25" May 28 15:10:30.284141 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.284117 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:30.289726 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.289708 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:30.297290 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.297272 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" May 28 15:10:30.304869 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.304848 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mndtq" May 28 15:10:30.312386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.312362 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jlwmt" May 28 15:10:30.319975 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.319956 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w7rh6" May 28 15:10:30.326521 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.326503 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" May 28 15:10:30.377592 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.377559 2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 28 15:10:30.571463 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.571442 2569 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 28 15:10:30.574294 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.574264 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09e9953_ec38_42d4_ac0b_353cfdd3683d.slice/crio-de8e0239844c0653509563236575b1194a047e4851d70120153272361e83d915 WatchSource:0}: Error finding container de8e0239844c0653509563236575b1194a047e4851d70120153272361e83d915: Status 404 returned error can't find the container with id de8e0239844c0653509563236575b1194a047e4851d70120153272361e83d915 May 28 15:10:30.575356 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.575334 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb848567f_fdd1_4b2c_82d2_f6692a9b4148.slice/crio-d612eb1386b8b7a8ef010d92f694f4a0520953edd426d219c96168de6782c2ff WatchSource:0}: Error finding container d612eb1386b8b7a8ef010d92f694f4a0520953edd426d219c96168de6782c2ff: Status 404 returned error can't find the container with id d612eb1386b8b7a8ef010d92f694f4a0520953edd426d219c96168de6782c2ff May 28 15:10:30.576632 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.576596 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded77076b_c4ca_4f11_934e_1f1ed7350613.slice/crio-46f30a5e64170b909d2ee9ee125530d289d545c1b99d79820f94616ef6387501 WatchSource:0}: Error finding container 46f30a5e64170b909d2ee9ee125530d289d545c1b99d79820f94616ef6387501: Status 404 returned error can't find the container with id 46f30a5e64170b909d2ee9ee125530d289d545c1b99d79820f94616ef6387501 May 28 15:10:30.578492 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.578401 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb0dbb2_e7bc_4fdf_9582_b5709495834a.slice/crio-a4f608289c85950cb8fb22319ea6e8abe4278b80068b766459aa9f8d19929b96 WatchSource:0}: Error finding container a4f608289c85950cb8fb22319ea6e8abe4278b80068b766459aa9f8d19929b96: Status 404 returned error can't find the container with id a4f608289c85950cb8fb22319ea6e8abe4278b80068b766459aa9f8d19929b96 May 28 15:10:30.579572 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.579547 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:30.579712 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.579697 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:30.579767 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.579759 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:31.579736377 +0000 UTC m=+4.082490531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:30.581082 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.581063 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5352d147_37e5_4291_9fe9_66faf7e057c2.slice/crio-45f5b21163e8640bb3ad1454a84d7c52b274ccb944e189cd2b0225e447d41848 WatchSource:0}: Error finding container 45f5b21163e8640bb3ad1454a84d7c52b274ccb944e189cd2b0225e447d41848: Status 404 returned error can't find the container with id 45f5b21163e8640bb3ad1454a84d7c52b274ccb944e189cd2b0225e447d41848 May 28 15:10:30.581999 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.581965 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cb0634_a302_4dfd_a99d_5de8107704dc.slice/crio-d2e017d8908e25b0d05d3ae1d8b22514c8c19b6968886b79fcd506a60bfc15c0 WatchSource:0}: Error finding container d2e017d8908e25b0d05d3ae1d8b22514c8c19b6968886b79fcd506a60bfc15c0: Status 404 returned error can't find the container with id d2e017d8908e25b0d05d3ae1d8b22514c8c19b6968886b79fcd506a60bfc15c0 May 28 15:10:30.583941 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.583919 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dd655e_c457_41ba_b059_59c44998d358.slice/crio-efe439ec364a863726f4468e6acd9c342909ddee389c218656ec63e4ae2ebdd8 WatchSource:0}: Error finding container efe439ec364a863726f4468e6acd9c342909ddee389c218656ec63e4ae2ebdd8: Status 404 returned error can't find the container with id efe439ec364a863726f4468e6acd9c342909ddee389c218656ec63e4ae2ebdd8 May 28 15:10:30.586112 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.586094 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f78757_e808_486a_9d6a_86aeccd930bb.slice/crio-64f9ada232a3062d23d7a17f152ccaea24c9a78424b0569393a69bef18a4cab7 WatchSource:0}: Error finding container 64f9ada232a3062d23d7a17f152ccaea24c9a78424b0569393a69bef18a4cab7: Status 404 returned error can't find the container with id 64f9ada232a3062d23d7a17f152ccaea24c9a78424b0569393a69bef18a4cab7 May 28 15:10:30.588290 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:10:30.588266 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fbf203_2dbe_401e_9d41_b9e841caa1e8.slice/crio-1e6144260c441bdfb67cb0faf54a2cc5061027294a5626ea2d86f4a6bc132783 WatchSource:0}: Error finding container 1e6144260c441bdfb67cb0faf54a2cc5061027294a5626ea2d86f4a6bc132783: Status 404 returned error can't find the container with id 1e6144260c441bdfb67cb0faf54a2cc5061027294a5626ea2d86f4a6bc132783 May 28 15:10:30.680449 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.680410 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:30.680607 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.680559 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 28 15:10:30.680607 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.680574 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 28 15:10:30.680607 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.680583 2569 projected.go:194] Error preparing data for projected volume kube-api-access-v4qkr for pod openshift-network-diagnostics/network-check-target-2vt4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:30.680759 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:30.680653 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr podName:e018896b-bc34-4bda-b761-752548a2cde1 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:31.680637296 +0000 UTC m=+4.183391450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4qkr" (UniqueName: "kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr") pod "network-check-target-2vt4s" (UID: "e018896b-bc34-4bda-b761-752548a2cde1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:30.998518 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.998477 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-27 15:05:28 +0000 UTC" deadline="2028-01-13 02:41:36.620585768 +0000 UTC" May 28 15:10:30.998518 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:30.998511 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14267h31m5.622077701s" May 28 15:10:31.069384 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.069303 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:31.069544 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:31.069441 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:31.081157 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.081120 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" event={"ID":"a61332edee42088905ed1b7a0d6356e8","Type":"ContainerStarted","Data":"a6ca5c6014a762b66fef12a59c9858bb7430fac592a8a17897821131f0271237"} May 28 15:10:31.084283 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.083785 2569 generic.go:358] "Generic (PLEG): container finished" podID="a47726c6874c43b16c1bd225ab259c42" containerID="372094f0e7a2e67ac2afdc1e49d69c9e91965e751e059f9d83a6aaf8f5dc8ed6" exitCode=0 May 28 15:10:31.084283 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.083940 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" event={"ID":"a47726c6874c43b16c1bd225ab259c42","Type":"ContainerDied","Data":"372094f0e7a2e67ac2afdc1e49d69c9e91965e751e059f9d83a6aaf8f5dc8ed6"} May 28 15:10:31.086668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.086595 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"1e6144260c441bdfb67cb0faf54a2cc5061027294a5626ea2d86f4a6bc132783"} May 28 15:10:31.089232 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.089205 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mndtq" event={"ID":"33dd655e-c457-41ba-b059-59c44998d358","Type":"ContainerStarted","Data":"efe439ec364a863726f4468e6acd9c342909ddee389c218656ec63e4ae2ebdd8"} May 28 15:10:31.091381 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.091354 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerStarted","Data":"d612eb1386b8b7a8ef010d92f694f4a0520953edd426d219c96168de6782c2ff"} May 28 15:10:31.093643 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.093535 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jlwmt" event={"ID":"83f78757-e808-486a-9d6a-86aeccd930bb","Type":"ContainerStarted","Data":"64f9ada232a3062d23d7a17f152ccaea24c9a78424b0569393a69bef18a4cab7"} May 28 15:10:31.096136 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.096113 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6d25" event={"ID":"e5cb0634-a302-4dfd-a99d-5de8107704dc","Type":"ContainerStarted","Data":"d2e017d8908e25b0d05d3ae1d8b22514c8c19b6968886b79fcd506a60bfc15c0"} May 28 15:10:31.097728 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.097676 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-98.ec2.internal" podStartSLOduration=2.097661218 podStartE2EDuration="2.097661218s" podCreationTimestamp="2026-05-28 15:10:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:10:31.096776586 +0000 UTC m=+3.599530748" watchObservedRunningTime="2026-05-28 15:10:31.097661218 +0000 UTC m=+3.600415380" May 28 15:10:31.104016 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.103949 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" event={"ID":"5352d147-37e5-4291-9fe9-66faf7e057c2","Type":"ContainerStarted","Data":"45f5b21163e8640bb3ad1454a84d7c52b274ccb944e189cd2b0225e447d41848"} May 28 15:10:31.106888 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.106799 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bg5rp" event={"ID":"8fb0dbb2-e7bc-4fdf-9582-b5709495834a","Type":"ContainerStarted","Data":"a4f608289c85950cb8fb22319ea6e8abe4278b80068b766459aa9f8d19929b96"} May 28 15:10:31.121525 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.121482 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" event={"ID":"ed77076b-c4ca-4f11-934e-1f1ed7350613","Type":"ContainerStarted","Data":"46f30a5e64170b909d2ee9ee125530d289d545c1b99d79820f94616ef6387501"} May 28 15:10:31.123322 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.123270 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w7rh6" event={"ID":"d09e9953-ec38-42d4-ac0b-353cfdd3683d","Type":"ContainerStarted","Data":"de8e0239844c0653509563236575b1194a047e4851d70120153272361e83d915"} May 28 15:10:31.588111 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.587500 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:31.588111 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:31.587667 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:31.588111 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:31.587733 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:33.587714767 +0000 UTC m=+6.090468922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:31.689408 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:31.688770 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:31.689408 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:31.688967 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 28 15:10:31.689408 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:31.688985 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 28 15:10:31.689408 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:31.688998 2569 projected.go:194] Error preparing data for projected volume kube-api-access-v4qkr for pod openshift-network-diagnostics/network-check-target-2vt4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:31.689408 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:31.689057 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr podName:e018896b-bc34-4bda-b761-752548a2cde1 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:33.689037656 +0000 UTC m=+6.191791807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4qkr" (UniqueName: "kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr") pod "network-check-target-2vt4s" (UID: "e018896b-bc34-4bda-b761-752548a2cde1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:32.071509 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:32.071424 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:32.071992 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:32.071563 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:32.134960 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:32.134833 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" event={"ID":"a47726c6874c43b16c1bd225ab259c42","Type":"ContainerStarted","Data":"6bb0200d57268e8efaa9007f4db2ff4cde2552832aa2dff4218694361d97ad63"} May 28 15:10:32.148212 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:32.148159 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-98.ec2.internal" podStartSLOduration=3.148140844 podStartE2EDuration="3.148140844s" podCreationTimestamp="2026-05-28 15:10:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:10:32.147750557 +0000 UTC m=+4.650504719" watchObservedRunningTime="2026-05-28 15:10:32.148140844 +0000 UTC m=+4.650895006" May 28 15:10:33.068986 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:33.068949 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:33.069164 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:33.069072 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:33.603521 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:33.602874 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:33.603521 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:33.603006 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:33.603521 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:33.603084 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:37.603063464 +0000 UTC m=+10.105817605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:33.704270 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:33.704224 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:33.704469 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:33.704445 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 28 15:10:33.704545 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:33.704471 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 28 15:10:33.704545 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:33.704485 2569 projected.go:194] Error preparing data for projected volume kube-api-access-v4qkr for pod openshift-network-diagnostics/network-check-target-2vt4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:33.704662 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:33.704558 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr podName:e018896b-bc34-4bda-b761-752548a2cde1 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:37.704537238 +0000 UTC m=+10.207291383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4qkr" (UniqueName: "kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr") pod "network-check-target-2vt4s" (UID: "e018896b-bc34-4bda-b761-752548a2cde1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:34.071159 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:34.069298 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:34.071309 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:34.071286 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:35.068627 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:35.068580 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:35.069160 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:35.068728 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:36.069658 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:36.069089 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:36.069658 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:36.069263 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:37.069470 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:37.069435 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:37.069670 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:37.069546 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:37.637329 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:37.637291 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:37.637728 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:37.637441 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:37.637728 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:37.637516 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:45.637497833 +0000 UTC m=+18.140251977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:37.738214 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:37.738111 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:37.738397 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:37.738346 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 28 15:10:37.738397 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:37.738372 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 28 15:10:37.738397 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:37.738385 2569 projected.go:194] Error preparing data for projected volume kube-api-access-v4qkr for pod openshift-network-diagnostics/network-check-target-2vt4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:37.738550 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:37.738454 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr podName:e018896b-bc34-4bda-b761-752548a2cde1 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:45.738435391 +0000 UTC m=+18.241189536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4qkr" (UniqueName: "kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr") pod "network-check-target-2vt4s" (UID: "e018896b-bc34-4bda-b761-752548a2cde1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:38.070314 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:38.070276 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:38.070492 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:38.070404 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:39.069148 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:39.069045 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:39.069571 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:39.069177 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:40.069049 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:40.069013 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:40.069225 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:40.069148 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:41.069102 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.068916 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:41.069285 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:41.069182 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:41.153941 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.153853 2569 generic.go:358] "Generic (PLEG): container finished" podID="b848567f-fdd1-4b2c-82d2-f6692a9b4148" containerID="feb65b1b0fd7344add7433f1aee02a5ad2130ab4655a1dd2379db29d843a9a79" exitCode=0 May 28 15:10:41.154093 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.153954 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerDied","Data":"feb65b1b0fd7344add7433f1aee02a5ad2130ab4655a1dd2379db29d843a9a79"} May 28 15:10:41.155865 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.155711 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jlwmt" event={"ID":"83f78757-e808-486a-9d6a-86aeccd930bb","Type":"ContainerStarted","Data":"b7678bf95e73a9054fb4add7a1fc2d7bd98c0acd0a55935044ab21dac40cd4d3"} May 28 15:10:41.157375 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.157352 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" event={"ID":"5352d147-37e5-4291-9fe9-66faf7e057c2","Type":"ContainerStarted","Data":"62246d0cfc8948ba86a0f4cab5f43abf6d323cfe9e0c0020bc0c0a0a0689aa56"} May 28 15:10:41.159272 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.159222 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bg5rp" event={"ID":"8fb0dbb2-e7bc-4fdf-9582-b5709495834a","Type":"ContainerStarted","Data":"556ed8aab7c65364edeee41db814444a0bf1e332a2d0a9a4715e62713eba0756"} May 28 15:10:41.161109 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.161091 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" event={"ID":"ed77076b-c4ca-4f11-934e-1f1ed7350613","Type":"ContainerStarted","Data":"4de77c20cce191b0d0b7a8240ce1ffb3c01a72c95670a0825524f94a03f38927"} May 28 15:10:41.162595 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.162573 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w7rh6" event={"ID":"d09e9953-ec38-42d4-ac0b-353cfdd3683d","Type":"ContainerStarted","Data":"e3735681ba950cdfb499e2578033ea614dce776029ca6dfbdf8458931bb37993"} May 28 15:10:41.196851 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.196800 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zsdb5" podStartSLOduration=3.557871028 podStartE2EDuration="13.196786644s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.582902077 +0000 UTC m=+3.085656216" lastFinishedPulling="2026-05-28 15:10:40.221817685 +0000 UTC m=+12.724571832" observedRunningTime="2026-05-28 15:10:41.196464164 +0000 UTC m=+13.699218325" watchObservedRunningTime="2026-05-28 15:10:41.196786644 +0000 UTC m=+13.699540825" May 28 15:10:41.197022 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.196916 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-jlwmt" podStartSLOduration=3.585465427 podStartE2EDuration="13.196909263s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.588432156 +0000 UTC m=+3.091186298" lastFinishedPulling="2026-05-28 15:10:40.199875985 +0000 UTC m=+12.702630134" observedRunningTime="2026-05-28 15:10:41.184337929 +0000 UTC m=+13.687092090" watchObservedRunningTime="2026-05-28 15:10:41.196909263 +0000 UTC m=+13.699663426" May 28 15:10:41.207806 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.207758 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bg5rp" podStartSLOduration=3.591316754 podStartE2EDuration="13.207742025s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.580482949 +0000 UTC m=+3.083237089" lastFinishedPulling="2026-05-28 15:10:40.196908206 +0000 UTC m=+12.699662360" observedRunningTime="2026-05-28 15:10:41.207402928 +0000 UTC m=+13.710157090" watchObservedRunningTime="2026-05-28 15:10:41.207742025 +0000 UTC m=+13.710496189" May 28 15:10:41.221503 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:41.221458 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w7rh6" podStartSLOduration=3.60939903 podStartE2EDuration="13.221444693s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.576124939 +0000 UTC m=+3.078879095" lastFinishedPulling="2026-05-28 15:10:40.188170611 +0000 UTC m=+12.690924758" observedRunningTime="2026-05-28 15:10:41.220868366 +0000 UTC m=+13.723622527" watchObservedRunningTime="2026-05-28 15:10:41.221444693 +0000 UTC m=+13.724198902" May 28 15:10:42.069431 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:42.069401 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:42.069933 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:42.069532 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:42.165907 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:42.165868 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l6d25" event={"ID":"e5cb0634-a302-4dfd-a99d-5de8107704dc","Type":"ContainerStarted","Data":"662c8d821361dd95eb31067fcf125a239068a0a9abd1aaba4ba11d7d90c198f9"} May 28 15:10:42.178269 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:42.178223 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l6d25" podStartSLOduration=4.561394041 podStartE2EDuration="14.178206332s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.583893873 +0000 UTC m=+3.086648014" lastFinishedPulling="2026-05-28 15:10:40.200706152 +0000 UTC m=+12.703460305" observedRunningTime="2026-05-28 15:10:42.177919939 +0000 UTC m=+14.680674100" watchObservedRunningTime="2026-05-28 15:10:42.178206332 +0000 UTC m=+14.680960494" May 28 15:10:42.373213 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:42.373125 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:42.373871 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:42.373849 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:43.069420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.069386 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:43.069597 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:43.069492 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:43.103223 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.103192 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bdr2w"] May 28 15:10:43.106044 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.106017 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.106169 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:43.106091 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:43.167450 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.167413 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:43.168377 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.168354 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bg5rp" May 28 15:10:43.180747 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.180715 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-dbus\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.180893 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.180802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-kubelet-config\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.180893 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.180828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.281719 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.281685 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-kubelet-config\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.281898 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.281750 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.281898 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.281819 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-kubelet-config\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.281898 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.281821 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-dbus\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.282060 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:43.281959 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 28 15:10:43.282060 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.282001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-dbus\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.282060 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:43.282018 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret podName:6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:43.781999022 +0000 UTC m=+16.284753165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret") pod "global-pull-secret-syncer-bdr2w" (UID: "6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7") : object "kube-system"/"original-pull-secret" not registered May 28 15:10:43.784696 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:43.784661 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:43.784860 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:43.784828 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 28 15:10:43.784922 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:43.784897 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret podName:6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:44.784876258 +0000 UTC m=+17.287630401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret") pod "global-pull-secret-syncer-bdr2w" (UID: "6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7") : object "kube-system"/"original-pull-secret" not registered May 28 15:10:44.069399 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:44.069320 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:44.069548 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:44.069452 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:44.792247 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:44.792049 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:44.792712 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:44.792186 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 28 15:10:44.792712 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:44.792400 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret podName:6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:46.792378564 +0000 UTC m=+19.295132705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret") pod "global-pull-secret-syncer-bdr2w" (UID: "6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7") : object "kube-system"/"original-pull-secret" not registered May 28 15:10:45.069039 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:45.068947 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:45.069207 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:45.068947 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:45.069207 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.069087 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:45.069301 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.069230 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:45.699305 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:45.699264 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:45.699554 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.699384 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:45.699554 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.699454 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:01.699436831 +0000 UTC m=+34.202190975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:10:45.800449 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:45.800405 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:45.800883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.800556 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 28 15:10:45.800883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.800575 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 28 15:10:45.800883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.800588 2569 projected.go:194] Error preparing data for projected volume kube-api-access-v4qkr for pod openshift-network-diagnostics/network-check-target-2vt4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:45.800883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:45.800668 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr podName:e018896b-bc34-4bda-b761-752548a2cde1 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:01.800649496 +0000 UTC m=+34.303403641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4qkr" (UniqueName: "kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr") pod "network-check-target-2vt4s" (UID: "e018896b-bc34-4bda-b761-752548a2cde1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:10:46.069242 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:46.069203 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:46.069410 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:46.069348 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:46.809151 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:46.809098 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:46.809601 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:46.809274 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 28 15:10:46.809601 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:46.809368 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret podName:6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:50.809348521 +0000 UTC m=+23.312102664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret") pod "global-pull-secret-syncer-bdr2w" (UID: "6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7") : object "kube-system"/"original-pull-secret" not registered May 28 15:10:47.069172 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:47.069076 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:47.069339 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:47.069076 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:47.069339 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:47.069224 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:47.069339 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:47.069304 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:48.069965 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:48.069925 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:48.070409 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:48.070056 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:49.068775 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:49.068741 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:49.068775 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:49.068779 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:49.068980 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:49.068856 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:49.068980 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:49.068923 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:49.539926 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:49.539885 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 28 15:10:50.042432 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.042089 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-28T15:10:49.539910309Z","UUID":"6d57d765-a1f9-4afe-84dc-a9275ac938c9","Handler":null,"Name":"","Endpoint":""} May 28 15:10:50.045925 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.045902 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 28 15:10:50.045925 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.045931 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 28 15:10:50.069043 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.069010 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:50.069223 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:50.069136 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:50.182829 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.182749 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:10:50.183115 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.183091 2569 generic.go:358] "Generic (PLEG): container finished" podID="67fbf203-2dbe-401e-9d41-b9e841caa1e8" containerID="902062071fe6820e63a882a1c814d613c6e027be5674ce966c5baeaa7ad49e6d" exitCode=1 May 28 15:10:50.183168 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.183140 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"00e690bf8303217f17ca33274f6a36709dd542bd5be43cd667e6382d396268e9"} May 28 15:10:50.183203 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.183173 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"f5699733c1c9e50209592eb511c6f4346bf3e3fd3cb3e6de7b1f2c4382d9ca01"} May 28 15:10:50.183203 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.183188 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"6b851dc558122019ce363ead224429fea416113bcb0b6616dddbf743297bb25e"} May 28 15:10:50.183203 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.183198 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"a4258e23ae6241290e54c29456b86537494039496adf50f1f77f5445e29d70da"} May 28 15:10:50.183304 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.183208 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerDied","Data":"902062071fe6820e63a882a1c814d613c6e027be5674ce966c5baeaa7ad49e6d"} May 28 15:10:50.183304 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.183222 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"b60489fe10d40ae44175d0dce1569753602b8c77e3961bc2d7eb0ec795026d5b"} May 28 15:10:50.184539 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.184508 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mndtq" event={"ID":"33dd655e-c457-41ba-b059-59c44998d358","Type":"ContainerStarted","Data":"aad1ee678bd875cf2c1b12566b738be91323f51551b255ec2943a84722c64122"} May 28 15:10:50.186038 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.186016 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" event={"ID":"ed77076b-c4ca-4f11-934e-1f1ed7350613","Type":"ContainerStarted","Data":"1afd1d2d3f3acc5a1153cf4de327f3f815065108ab1d9b830f980654557442ec"} May 28 15:10:50.210259 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.210181 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-mndtq" podStartSLOduration=3.396705801 podStartE2EDuration="22.210165833s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.585571878 +0000 UTC m=+3.088326017" lastFinishedPulling="2026-05-28 15:10:49.399031896 +0000 UTC m=+21.901786049" observedRunningTime="2026-05-28 15:10:50.209863725 +0000 UTC m=+22.712617886" watchObservedRunningTime="2026-05-28 15:10:50.210165833 +0000 UTC m=+22.712919994" May 28 15:10:50.837432 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:50.837396 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:50.838047 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:50.837536 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 28 15:10:50.838047 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:50.837599 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret podName:6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7 nodeName:}" failed. No retries permitted until 2026-05-28 15:10:58.837585152 +0000 UTC m=+31.340339301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret") pod "global-pull-secret-syncer-bdr2w" (UID: "6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7") : object "kube-system"/"original-pull-secret" not registered May 28 15:10:51.069011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:51.068978 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:51.069011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:51.069007 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:51.069247 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:51.069101 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:51.069247 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:51.069216 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:51.190513 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:51.190469 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" event={"ID":"ed77076b-c4ca-4f11-934e-1f1ed7350613","Type":"ContainerStarted","Data":"43447762d30d286c454281cb1aac6083cacede0fdacb3b86a2b36eb1875c1f7c"} May 28 15:10:52.068801 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:52.068771 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:52.069189 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:52.068882 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:52.194494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:52.194465 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:10:52.194821 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:52.194793 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"ed5e21a6f8904091e50dce3cef23ce83a95c9af656c2ddf7a3155073cd8e1061"} May 28 15:10:53.068955 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:53.068920 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:53.069418 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:53.068920 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:53.069418 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:53.069070 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:53.069418 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:53.069172 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:54.068916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:54.068882 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:54.069102 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:54.069024 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:54.202411 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:54.202178 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:10:54.204678 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:54.204649 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerStarted","Data":"b57d1ed4f3c5cce8d63fd0404899eb2e4a203bb4b410fa3dfe98ad33fc8a2115"} May 28 15:10:54.230858 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:54.230696 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sq46t" podStartSLOduration=6.08678664 podStartE2EDuration="26.230675634s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.578305434 +0000 UTC m=+3.081059589" lastFinishedPulling="2026-05-28 15:10:50.722194441 +0000 UTC m=+23.224948583" observedRunningTime="2026-05-28 15:10:51.206991347 +0000 UTC m=+23.709745519" watchObservedRunningTime="2026-05-28 15:10:54.230675634 +0000 UTC m=+26.733429797" May 28 15:10:55.069265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.069235 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:55.069667 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.069235 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:55.069667 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:55.069328 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:55.069667 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:55.069409 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:55.209497 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.209471 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:10:55.209822 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.209799 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"71d163eeeb668796096110ab81439599e4f65ec7bba9f7b3bdfcdc587aa24fe2"} May 28 15:10:55.210142 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.210112 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:55.210295 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.210277 2569 scope.go:117] "RemoveContainer" containerID="902062071fe6820e63a882a1c814d613c6e027be5674ce966c5baeaa7ad49e6d" May 28 15:10:55.211410 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.211387 2569 generic.go:358] "Generic (PLEG): container finished" podID="b848567f-fdd1-4b2c-82d2-f6692a9b4148" containerID="b57d1ed4f3c5cce8d63fd0404899eb2e4a203bb4b410fa3dfe98ad33fc8a2115" exitCode=0 May 28 15:10:55.211491 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.211421 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerDied","Data":"b57d1ed4f3c5cce8d63fd0404899eb2e4a203bb4b410fa3dfe98ad33fc8a2115"} May 28 15:10:55.226514 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:55.226487 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:56.069241 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.069146 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:56.069383 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:56.069258 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:56.106134 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.106103 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ckhtc"] May 28 15:10:56.109226 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.109202 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vt4s"] May 28 15:10:56.109355 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.109319 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:56.109476 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:56.109415 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:56.110109 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.110089 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdr2w"] May 28 15:10:56.110215 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.110180 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:56.110278 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:56.110258 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:56.215983 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.215956 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:10:56.216339 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.216310 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" event={"ID":"67fbf203-2dbe-401e-9d41-b9e841caa1e8","Type":"ContainerStarted","Data":"1bb717546522180476e339666430f1ca792d766d2eca1c5f8ebd5e857c60d92b"} May 28 15:10:56.216668 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.216641 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:56.216766 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.216673 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:56.218248 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.218224 2569 generic.go:358] "Generic (PLEG): container finished" podID="b848567f-fdd1-4b2c-82d2-f6692a9b4148" containerID="16edfc2cc05657f117109bb47ecd91a49e12627e3611f5b8edfc7f883ee3712b" exitCode=0 May 28 15:10:56.218349 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.218311 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerDied","Data":"16edfc2cc05657f117109bb47ecd91a49e12627e3611f5b8edfc7f883ee3712b"} May 28 15:10:56.218349 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.218342 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:56.218469 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:56.218433 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:56.231965 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.231940 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:10:56.243815 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:56.243766 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" podStartSLOduration=9.394073344 podStartE2EDuration="28.243753581s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.590513856 +0000 UTC m=+3.093267997" lastFinishedPulling="2026-05-28 15:10:49.44019409 +0000 UTC m=+21.942948234" observedRunningTime="2026-05-28 15:10:56.243252435 +0000 UTC m=+28.746006596" watchObservedRunningTime="2026-05-28 15:10:56.243753581 +0000 UTC m=+28.746507739" May 28 15:10:57.222065 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:57.222030 2569 generic.go:358] "Generic (PLEG): container finished" podID="b848567f-fdd1-4b2c-82d2-f6692a9b4148" containerID="e67dbda8d4dd56342f92b7dc9d9f13dd2beee9c815a1c7bf2e462abf83659be4" exitCode=0 May 28 15:10:57.222494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:57.222121 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerDied","Data":"e67dbda8d4dd56342f92b7dc9d9f13dd2beee9c815a1c7bf2e462abf83659be4"} May 28 15:10:58.069210 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:58.069182 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:10:58.069402 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:58.069264 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:10:58.069402 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:58.069278 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:58.069402 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:58.069364 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:10:58.069585 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:58.069414 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:10:58.069585 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:58.069524 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:10:58.902889 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:10:58.902715 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:10:58.902889 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:58.902884 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 28 15:10:58.903421 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:10:58.902964 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret podName:6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:14.902943151 +0000 UTC m=+47.405697294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret") pod "global-pull-secret-syncer-bdr2w" (UID: "6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7") : object "kube-system"/"original-pull-secret" not registered May 28 15:11:00.069030 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:00.068990 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:00.069564 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:00.069124 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:11:00.069564 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:00.069343 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:11:00.069564 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:00.069446 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:11:00.069564 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:00.068990 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:11:00.069564 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:00.069555 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:11:01.722139 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:01.722099 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:11:01.722733 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:01.722265 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:11:01.722733 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:01.722337 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:33.722318269 +0000 UTC m=+66.225072415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : object "openshift-multus"/"metrics-daemon-secret" not registered May 28 15:11:01.823032 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:01.822991 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:01.823201 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:01.823151 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 28 15:11:01.823201 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:01.823177 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 28 15:11:01.823201 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:01.823190 2569 projected.go:194] Error preparing data for projected volume kube-api-access-v4qkr for pod openshift-network-diagnostics/network-check-target-2vt4s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:11:01.823349 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:01.823255 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr podName:e018896b-bc34-4bda-b761-752548a2cde1 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:33.823235033 +0000 UTC m=+66.325989174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v4qkr" (UniqueName: "kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr") pod "network-check-target-2vt4s" (UID: "e018896b-bc34-4bda-b761-752548a2cde1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 28 15:11:02.069424 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.069390 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:02.069607 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.069390 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:11:02.069607 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.069487 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2vt4s" podUID="e018896b-bc34-4bda-b761-752548a2cde1" May 28 15:11:02.069607 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.069581 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bdr2w" podUID="6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7" May 28 15:11:02.069607 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.069390 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:11:02.069847 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.069714 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:11:02.269033 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.268997 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-98.ec2.internal" event="NodeReady" May 28 15:11:02.269205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.269157 2569 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 28 15:11:02.307564 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.307428 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp"] May 28 15:11:02.313630 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.313594 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n"] May 28 15:11:02.313784 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.313770 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.316744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.316416 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" May 28 15:11:02.316744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.316421 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" May 28 15:11:02.316744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.316467 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" May 28 15:11:02.316744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.316499 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" May 28 15:11:02.316744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.316531 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" May 28 15:11:02.316744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.316466 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" May 28 15:11:02.316744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.316736 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" May 28 15:11:02.317969 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.317950 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk"] May 28 15:11:02.318137 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.318122 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.320448 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.320386 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jjtgj\"" May 28 15:11:02.320551 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.320490 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" May 28 15:11:02.322787 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.322644 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dd7bd75b8-mqcfl"] May 28 15:11:02.328070 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.326046 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.330903 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.330590 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" May 28 15:11:02.336688 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.336664 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ggbmj"] May 28 15:11:02.336942 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.336926 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.340998 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.340903 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 28 15:11:02.340998 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.340928 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 28 15:11:02.341233 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.341218 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-glc25\"" May 28 15:11:02.341291 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.341266 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 28 15:11:02.341455 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.341439 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7qp66"] May 28 15:11:02.341600 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.341584 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:02.344085 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.343965 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 28 15:11:02.344733 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.344205 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 28 15:11:02.344733 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.344236 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ljmrm\"" May 28 15:11:02.344733 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.344411 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 28 15:11:02.345470 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.345439 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n"] May 28 15:11:02.345470 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.345470 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp"] May 28 15:11:02.345669 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.345486 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk"] May 28 15:11:02.345669 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.345500 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd7bd75b8-mqcfl"] May 28 15:11:02.345669 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.345510 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7qp66"] May 28 15:11:02.345669 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.345526 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggbmj"] May 28 15:11:02.345900 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.345681 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.347022 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.347005 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 28 15:11:02.348289 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.348271 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 28 15:11:02.348395 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.348379 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8f6nh\"" May 28 15:11:02.348875 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.348832 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 28 15:11:02.431958 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.431923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-hub\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.432152 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.431967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.432152 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.431999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d2a06f-69eb-4689-99cf-26f2aea43403-ca-trust-extracted\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.432152 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-bound-sa-token\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.432152 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432094 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d53a9613-b036-4eab-a8bb-7a7713ad1211-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fff6d8f4-brr5n\" (UID: \"d53a9613-b036-4eab-a8bb-7a7713ad1211\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.432360 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjgv\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-kube-api-access-5kjgv\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.432360 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snp2v\" (UniqueName: \"kubernetes.io/projected/d53a9613-b036-4eab-a8bb-7a7713ad1211-kube-api-access-snp2v\") pod \"managed-serviceaccount-addon-agent-76fff6d8f4-brr5n\" (UID: \"d53a9613-b036-4eab-a8bb-7a7713ad1211\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.432360 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432246 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-trusted-ca\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.432360 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-tmp\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.432360 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432341 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-klusterlet-config\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.432597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432397 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-installation-pull-secrets\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.432597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgb59\" (UniqueName: \"kubernetes.io/projected/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-kube-api-access-fgb59\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.432597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4dc93c-2918-4976-a0a3-800fd9078a13-tmp-dir\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.432597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-certificates\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.432597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5dj\" (UniqueName: \"kubernetes.io/projected/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-kube-api-access-8f5dj\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:02.432597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wk6\" (UniqueName: \"kubernetes.io/projected/df4dc93c-2918-4976-a0a3-800fd9078a13-kube-api-access-s5wk6\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.432597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432557 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df4dc93c-2918-4976-a0a3-800fd9078a13-config-volume\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432688 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-ca\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432737 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/252cc408-818d-4952-b377-04d7069c75f2-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432904 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4fpf\" (UniqueName: \"kubernetes.io/projected/252cc408-818d-4952-b377-04d7069c75f2-kube-api-access-x4fpf\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.432971 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.432931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-image-registry-private-configuration\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.533358 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533312 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:02.533358 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533375 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df4dc93c-2918-4976-a0a3-800fd9078a13-config-volume\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.533358 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533401 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-ca\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533429 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533463 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533487 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/252cc408-818d-4952-b377-04d7069c75f2-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533530 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533558 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x4fpf\" (UniqueName: \"kubernetes.io/projected/252cc408-818d-4952-b377-04d7069c75f2-kube-api-access-x4fpf\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533584 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-image-registry-private-configuration\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533643 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-hub\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.533708 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.533664 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:11:02.534116 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.533732 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:11:02.534116 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.533741 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:03.033720703 +0000 UTC m=+35.536474846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:11:02.534116 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.533744 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:11:02.534116 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.533824 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:03.033812698 +0000 UTC m=+35.536566847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:11:02.534116 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534070 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df4dc93c-2918-4976-a0a3-800fd9078a13-config-volume\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.534402 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.534174 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:11:02.534402 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:02.534226 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:11:03.034210281 +0000 UTC m=+35.536964439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:11:02.534402 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.533666 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.534402 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534268 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d2a06f-69eb-4689-99cf-26f2aea43403-ca-trust-extracted\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.534402 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534307 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-bound-sa-token\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.534402 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534352 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d53a9613-b036-4eab-a8bb-7a7713ad1211-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fff6d8f4-brr5n\" (UID: \"d53a9613-b036-4eab-a8bb-7a7713ad1211\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534667 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjgv\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-kube-api-access-5kjgv\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534709 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-snp2v\" (UniqueName: \"kubernetes.io/projected/d53a9613-b036-4eab-a8bb-7a7713ad1211-kube-api-access-snp2v\") pod \"managed-serviceaccount-addon-agent-76fff6d8f4-brr5n\" (UID: \"d53a9613-b036-4eab-a8bb-7a7713ad1211\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534738 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-trusted-ca\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534765 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-tmp\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534794 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-klusterlet-config\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534826 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-installation-pull-secrets\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534858 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgb59\" (UniqueName: \"kubernetes.io/projected/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-kube-api-access-fgb59\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534892 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4dc93c-2918-4976-a0a3-800fd9078a13-tmp-dir\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534924 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-certificates\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534948 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5dj\" (UniqueName: \"kubernetes.io/projected/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-kube-api-access-8f5dj\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:02.534989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.534977 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wk6\" (UniqueName: \"kubernetes.io/projected/df4dc93c-2918-4976-a0a3-800fd9078a13-kube-api-access-s5wk6\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.536492 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.535155 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/252cc408-818d-4952-b377-04d7069c75f2-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.536492 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.535411 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d2a06f-69eb-4689-99cf-26f2aea43403-ca-trust-extracted\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.536492 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.536115 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-tmp\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.536492 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.536396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-trusted-ca\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.536719 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.536693 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4dc93c-2918-4976-a0a3-800fd9078a13-tmp-dir\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.537732 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.537179 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-certificates\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.539465 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.539149 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-klusterlet-config\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.539465 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.539250 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-image-registry-private-configuration\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.540013 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.539989 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d53a9613-b036-4eab-a8bb-7a7713ad1211-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fff6d8f4-brr5n\" (UID: \"d53a9613-b036-4eab-a8bb-7a7713ad1211\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.540119 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.540046 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-installation-pull-secrets\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.540119 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.540069 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-ca\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.540239 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.540110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-hub\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.540389 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.540366 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.541503 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.541482 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/252cc408-818d-4952-b377-04d7069c75f2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.543641 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.543570 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4fpf\" (UniqueName: \"kubernetes.io/projected/252cc408-818d-4952-b377-04d7069c75f2-kube-api-access-x4fpf\") pod \"cluster-proxy-proxy-agent-d5f67fd5d-v6kkp\" (UID: \"252cc408-818d-4952-b377-04d7069c75f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.544350 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.544287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snp2v\" (UniqueName: \"kubernetes.io/projected/d53a9613-b036-4eab-a8bb-7a7713ad1211-kube-api-access-snp2v\") pod \"managed-serviceaccount-addon-agent-76fff6d8f4-brr5n\" (UID: \"d53a9613-b036-4eab-a8bb-7a7713ad1211\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.544350 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.544327 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wk6\" (UniqueName: \"kubernetes.io/projected/df4dc93c-2918-4976-a0a3-800fd9078a13-kube-api-access-s5wk6\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:02.544857 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.544826 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-bound-sa-token\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.545193 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.545171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgb59\" (UniqueName: \"kubernetes.io/projected/eca9c31d-2a24-45f0-8e9a-1cb49b550e4c-kube-api-access-fgb59\") pod \"klusterlet-addon-workmgr-695665dd76-gg6vk\" (UID: \"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:02.545329 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.545311 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjgv\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-kube-api-access-5kjgv\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:02.547023 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.546990 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5dj\" (UniqueName: \"kubernetes.io/projected/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-kube-api-access-8f5dj\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:02.646927 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.646835 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:11:02.652751 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.652724 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:11:02.670588 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:02.670555 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:03.040129 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:03.040100 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:03.040140 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:03.040201 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:03.040269 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:03.040317 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:03.040332 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:03.040341 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:11:04.040321838 +0000 UTC m=+36.543075978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:03.040344 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:03.040386 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:04.040372051 +0000 UTC m=+36.543126201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:11:03.040791 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:03.040405 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:04.040395582 +0000 UTC m=+36.543149730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:11:03.495512 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:03.495295 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n"] May 28 15:11:03.498032 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:03.498007 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk"] May 28 15:11:03.501110 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:03.501085 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp"] May 28 15:11:03.522229 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:11:03.522187 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd53a9613_b036_4eab_a8bb_7a7713ad1211.slice/crio-481e1dca7526effe6e442c6782c9d2a65e8c3e9aac9a86c4dcbacf1ceafc58e3 WatchSource:0}: Error finding container 481e1dca7526effe6e442c6782c9d2a65e8c3e9aac9a86c4dcbacf1ceafc58e3: Status 404 returned error can't find the container with id 481e1dca7526effe6e442c6782c9d2a65e8c3e9aac9a86c4dcbacf1ceafc58e3 May 28 15:11:03.522633 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:11:03.522600 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca9c31d_2a24_45f0_8e9a_1cb49b550e4c.slice/crio-b474630e02d4083bf8beee872abf190c602b5eb755cf3c413b38a7acf6a6c3b2 WatchSource:0}: Error finding container b474630e02d4083bf8beee872abf190c602b5eb755cf3c413b38a7acf6a6c3b2: Status 404 returned error can't find the container with id b474630e02d4083bf8beee872abf190c602b5eb755cf3c413b38a7acf6a6c3b2 May 28 15:11:03.523567 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:11:03.523483 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod252cc408_818d_4952_b377_04d7069c75f2.slice/crio-56e36ed1bc2f897b469fcdcb8622e9c80020a22c47223d17754470d36afa53ab WatchSource:0}: Error finding container 56e36ed1bc2f897b469fcdcb8622e9c80020a22c47223d17754470d36afa53ab: Status 404 returned error can't find the container with id 56e36ed1bc2f897b469fcdcb8622e9c80020a22c47223d17754470d36afa53ab May 28 15:11:04.050382 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.050340 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.050395 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.050455 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:04.050498 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:04.050554 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:04.050584 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:04.050562 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:11:06.050542793 +0000 UTC m=+38.553296950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:04.050596 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:04.050635 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:06.050601383 +0000 UTC m=+38.553355523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:11:04.050814 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:04.050656 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:06.050644441 +0000 UTC m=+38.553398589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:11:04.068749 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.068712 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:11:04.068925 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.068712 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:04.068925 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.068712 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:11:04.071136 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.071117 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 28 15:11:04.071393 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.071137 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 28 15:11:04.071393 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.071196 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-49m45\"" May 28 15:11:04.072149 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.072124 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 28 15:11:04.072246 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.072135 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 28 15:11:04.072246 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.072184 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwwqm\"" May 28 15:11:04.237473 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.237429 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" event={"ID":"252cc408-818d-4952-b377-04d7069c75f2","Type":"ContainerStarted","Data":"56e36ed1bc2f897b469fcdcb8622e9c80020a22c47223d17754470d36afa53ab"} May 28 15:11:04.239767 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.239737 2569 generic.go:358] "Generic (PLEG): container finished" podID="b848567f-fdd1-4b2c-82d2-f6692a9b4148" containerID="9910afc7a0a9639bba588580137217c04233251b3c030bcc25d4c40b3bf46a8d" exitCode=0 May 28 15:11:04.239928 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.239814 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerDied","Data":"9910afc7a0a9639bba588580137217c04233251b3c030bcc25d4c40b3bf46a8d"} May 28 15:11:04.240829 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.240806 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" event={"ID":"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c","Type":"ContainerStarted","Data":"b474630e02d4083bf8beee872abf190c602b5eb755cf3c413b38a7acf6a6c3b2"} May 28 15:11:04.241860 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:04.241838 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" event={"ID":"d53a9613-b036-4eab-a8bb-7a7713ad1211","Type":"ContainerStarted","Data":"481e1dca7526effe6e442c6782c9d2a65e8c3e9aac9a86c4dcbacf1ceafc58e3"} May 28 15:11:05.253216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:05.253089 2569 generic.go:358] "Generic (PLEG): container finished" podID="b848567f-fdd1-4b2c-82d2-f6692a9b4148" containerID="71e2a52c69b8b9ac738dd3cf12af98b60e0cf03216dd3df532e49ec7aef2b9c4" exitCode=0 May 28 15:11:05.253216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:05.253147 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerDied","Data":"71e2a52c69b8b9ac738dd3cf12af98b60e0cf03216dd3df532e49ec7aef2b9c4"} May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:06.070632 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:06.070746 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:06.070773 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:06.070746 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:06.070837 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:10.070817567 +0000 UTC m=+42.573571711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:06.070877 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:06.070965 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:06.071015 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:06.071111 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:11:10.07103291 +0000 UTC m=+42.573787050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:11:06.071193 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:06.071147 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:10.071123564 +0000 UTC m=+42.573877711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:11:06.260301 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:06.260263 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" event={"ID":"b848567f-fdd1-4b2c-82d2-f6692a9b4148","Type":"ContainerStarted","Data":"a54b786e515be374fffa2c545adeb5f67db2b66cd62fa23a3d3968556b3faab4"} May 28 15:11:06.285253 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:06.285143 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8vmcs" podStartSLOduration=5.308314751 podStartE2EDuration="38.285123021s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:10:30.577335393 +0000 UTC m=+3.080089533" lastFinishedPulling="2026-05-28 15:11:03.554143652 +0000 UTC m=+36.056897803" observedRunningTime="2026-05-28 15:11:06.283161435 +0000 UTC m=+38.785915598" watchObservedRunningTime="2026-05-28 15:11:06.285123021 +0000 UTC m=+38.787877184" May 28 15:11:10.104510 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:10.104302 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:10.104575 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:10.104471 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:10.104607 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:10.104609 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:10.104688 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:18.104668274 +0000 UTC m=+50.607422419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:10.104700 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:10.104755 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:10.104782 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:18.104769399 +0000 UTC m=+50.607523550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:11:10.105084 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:10.104826 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:11:18.104805997 +0000 UTC m=+50.607560144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:11:12.274427 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:12.274340 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" event={"ID":"d53a9613-b036-4eab-a8bb-7a7713ad1211","Type":"ContainerStarted","Data":"401e4b0e326636353ced98eb8d5a75581664260ad95ede8912e3845f25c0310d"} May 28 15:11:12.275600 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:12.275580 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" event={"ID":"252cc408-818d-4952-b377-04d7069c75f2","Type":"ContainerStarted","Data":"c457e8709c40a236421589d074d5283b75011439f70abf68c9486037933a3c8e"} May 28 15:11:12.276675 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:12.276651 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" event={"ID":"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c","Type":"ContainerStarted","Data":"a4739b2006f795764a959a53310bc886d365d796d13884b5efddd84260624064"} May 28 15:11:12.276900 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:12.276885 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:12.278633 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:12.278594 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:11:12.290002 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:12.289960 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" podStartSLOduration=20.905800329 podStartE2EDuration="29.289950196s" podCreationTimestamp="2026-05-28 15:10:43 +0000 UTC" firstStartedPulling="2026-05-28 15:11:03.529063067 +0000 UTC m=+36.031817207" lastFinishedPulling="2026-05-28 15:11:11.913212922 +0000 UTC m=+44.415967074" observedRunningTime="2026-05-28 15:11:12.289743451 +0000 UTC m=+44.792497613" watchObservedRunningTime="2026-05-28 15:11:12.289950196 +0000 UTC m=+44.792704357" May 28 15:11:12.306441 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:12.306394 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" podStartSLOduration=21.395782 podStartE2EDuration="29.30638059s" podCreationTimestamp="2026-05-28 15:10:43 +0000 UTC" firstStartedPulling="2026-05-28 15:11:03.529076446 +0000 UTC m=+36.031830589" lastFinishedPulling="2026-05-28 15:11:11.439675035 +0000 UTC m=+43.942429179" observedRunningTime="2026-05-28 15:11:12.306156637 +0000 UTC m=+44.808910800" watchObservedRunningTime="2026-05-28 15:11:12.30638059 +0000 UTC m=+44.809134754" May 28 15:11:14.944952 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:14.944910 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:11:14.948831 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:14.948809 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7-original-pull-secret\") pod \"global-pull-secret-syncer-bdr2w\" (UID: \"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7\") " pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:11:15.191221 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:15.191174 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bdr2w" May 28 15:11:15.285151 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:15.284947 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" event={"ID":"252cc408-818d-4952-b377-04d7069c75f2","Type":"ContainerStarted","Data":"0e9e0adf6acbe3077cf56cd92aab01b7a5ceb5385817856b1b53df5ef86e1ecd"} May 28 15:11:15.285151 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:15.285130 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" event={"ID":"252cc408-818d-4952-b377-04d7069c75f2","Type":"ContainerStarted","Data":"5f04b5b408e658dc6ab1076c99bec8980b688d4592b7c7788c9c644428f28927"} May 28 15:11:15.303928 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:15.303885 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" podStartSLOduration=21.118578002 podStartE2EDuration="32.303871126s" podCreationTimestamp="2026-05-28 15:10:43 +0000 UTC" firstStartedPulling="2026-05-28 15:11:03.529020634 +0000 UTC m=+36.031774777" lastFinishedPulling="2026-05-28 15:11:14.714313759 +0000 UTC m=+47.217067901" observedRunningTime="2026-05-28 15:11:15.303597661 +0000 UTC m=+47.806351824" watchObservedRunningTime="2026-05-28 15:11:15.303871126 +0000 UTC m=+47.806625286" May 28 15:11:15.315180 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:15.315154 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bdr2w"] May 28 15:11:15.319112 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:11:15.319085 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b36d81d_a1fb_41b3_bf1f_52cf9e95b2c7.slice/crio-479ae333eac9a95cc7b8199cbdee0d0a44961f1c5b61be3cbeb0ed81a2f1ca8e WatchSource:0}: Error finding container 479ae333eac9a95cc7b8199cbdee0d0a44961f1c5b61be3cbeb0ed81a2f1ca8e: Status 404 returned error can't find the container with id 479ae333eac9a95cc7b8199cbdee0d0a44961f1c5b61be3cbeb0ed81a2f1ca8e May 28 15:11:16.287878 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:16.287833 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdr2w" event={"ID":"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7","Type":"ContainerStarted","Data":"479ae333eac9a95cc7b8199cbdee0d0a44961f1c5b61be3cbeb0ed81a2f1ca8e"} May 28 15:11:18.169703 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:18.169664 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:18.169753 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:18.169816 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:18.169869 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:18.169870 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:18.169888 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:18.169891 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:34.169867463 +0000 UTC m=+66.672621605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:18.169921 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:18.169926 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:11:34.169915202 +0000 UTC m=+66.672669341 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:11:18.170218 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:18.169961 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:11:34.169950213 +0000 UTC m=+66.672704357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:11:20.298222 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:20.298186 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bdr2w" event={"ID":"6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7","Type":"ContainerStarted","Data":"e30c29b44cc715d96ded8faea03f8fa3a7ddbb4d9c4d6fdfecdb515e53bbfb8f"} May 28 15:11:20.312576 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:20.312525 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bdr2w" podStartSLOduration=32.865683662 podStartE2EDuration="37.312508665s" podCreationTimestamp="2026-05-28 15:10:43 +0000 UTC" firstStartedPulling="2026-05-28 15:11:15.320771523 +0000 UTC m=+47.823525662" lastFinishedPulling="2026-05-28 15:11:19.767596518 +0000 UTC m=+52.270350665" observedRunningTime="2026-05-28 15:11:20.31217786 +0000 UTC m=+52.814932021" watchObservedRunningTime="2026-05-28 15:11:20.312508665 +0000 UTC m=+52.815262826" May 28 15:11:28.245491 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:28.245454 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gx9nw" May 28 15:11:33.794302 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:33.794256 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:11:33.797142 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:33.797122 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 28 15:11:33.805085 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:33.805063 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 28 15:11:33.805186 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:33.805122 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:37.805105684 +0000 UTC m=+130.307859829 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : secret "metrics-daemon-secret" not found May 28 15:11:33.895289 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:33.895235 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:33.898181 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:33.898161 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 28 15:11:33.908800 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:33.908775 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 28 15:11:33.920764 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:33.920743 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/e018896b-bc34-4bda-b761-752548a2cde1-kube-api-access-v4qkr\") pod \"network-check-target-2vt4s\" (UID: \"e018896b-bc34-4bda-b761-752548a2cde1\") " pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:34.089058 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:34.088975 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwwqm\"" May 28 15:11:34.096262 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:34.096237 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:34.197491 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:34.197463 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:11:34.197674 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:34.197540 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:11:34.197674 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:34.197609 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:11:34.197674 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:34.197609 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:11:34.197674 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:34.197674 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:11:34.197883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:34.197702 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:11:34.197883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:34.197738 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:06.1977199 +0000 UTC m=+98.700474039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:11:34.197883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:34.197761 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:12:06.197747743 +0000 UTC m=+98.700501882 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:11:34.197883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:34.197707 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:11:34.197883 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:11:34.197787 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:12:06.197780922 +0000 UTC m=+98.700535060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:11:34.213262 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:34.213234 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2vt4s"] May 28 15:11:34.330561 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:34.330526 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vt4s" event={"ID":"e018896b-bc34-4bda-b761-752548a2cde1","Type":"ContainerStarted","Data":"e5f4ffba90e539b5a9934c2a2736d681d5a44177c02c4fb6f58e649870eacf4e"} May 28 15:11:37.339110 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:37.339071 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2vt4s" event={"ID":"e018896b-bc34-4bda-b761-752548a2cde1","Type":"ContainerStarted","Data":"7fe8f4cc9f66377299ca2ffeeaeb119bdd82a9d447763a124571e6d90e29f3c7"} May 28 15:11:37.339522 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:37.339196 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:11:37.354537 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:11:37.354491 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2vt4s" podStartSLOduration=66.642314489 podStartE2EDuration="1m9.354478774s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:11:34.219524595 +0000 UTC m=+66.722278734" lastFinishedPulling="2026-05-28 15:11:36.931688875 +0000 UTC m=+69.434443019" observedRunningTime="2026-05-28 15:11:37.353588622 +0000 UTC m=+69.856342783" watchObservedRunningTime="2026-05-28 15:11:37.354478774 +0000 UTC m=+69.857232951" May 28 15:12:06.248708 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:12:06.248672 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:12:06.248741 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:12:06.248772 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:06.248820 2569 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:06.248841 2569 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dd7bd75b8-mqcfl: secret "image-registry-tls" not found May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:06.248873 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:06.248902 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls podName:f7d2a06f-69eb-4689-99cf-26f2aea43403 nodeName:}" failed. No retries permitted until 2026-05-28 15:13:10.248885314 +0000 UTC m=+162.751639452 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls") pod "image-registry-6dd7bd75b8-mqcfl" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403") : secret "image-registry-tls" not found May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:06.248924 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls podName:df4dc93c-2918-4976-a0a3-800fd9078a13 nodeName:}" failed. No retries permitted until 2026-05-28 15:13:10.248912556 +0000 UTC m=+162.751666695 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls") pod "dns-default-7qp66" (UID: "df4dc93c-2918-4976-a0a3-800fd9078a13") : secret "dns-default-metrics-tls" not found May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:06.248873 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 28 15:12:06.249052 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:06.248951 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert podName:f7574eb4-3b1e-478d-9f61-dbf80303c7ea nodeName:}" failed. No retries permitted until 2026-05-28 15:13:10.248943142 +0000 UTC m=+162.751697282 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert") pod "ingress-canary-ggbmj" (UID: "f7574eb4-3b1e-478d-9f61-dbf80303c7ea") : secret "canary-serving-cert" not found May 28 15:12:08.343572 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:12:08.343539 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2vt4s" May 28 15:12:37.881769 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:12:37.881730 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:12:37.882192 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:37.881866 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 28 15:12:37.882192 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:12:37.881931 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs podName:70961166-700d-40ad-867d-dcdf297e6704 nodeName:}" failed. No retries permitted until 2026-05-28 15:14:39.881915833 +0000 UTC m=+252.384669971 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs") pod "network-metrics-daemon-ckhtc" (UID: "70961166-700d-40ad-867d-dcdf297e6704") : secret "metrics-daemon-secret" not found May 28 15:12:51.406717 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:12:51.406690 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jlwmt_83f78757-e808-486a-9d6a-86aeccd930bb/dns-node-resolver/0.log" May 28 15:12:52.407364 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:12:52.407335 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w7rh6_d09e9953-ec38-42d4-ac0b-353cfdd3683d/node-ca/0.log" May 28 15:13:05.378337 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:05.378292 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" May 28 15:13:05.384387 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:05.384360 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-ggbmj" podUID="f7574eb4-3b1e-478d-9f61-dbf80303c7ea" May 28 15:13:05.391650 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:05.391602 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7qp66" podUID="df4dc93c-2918-4976-a0a3-800fd9078a13" May 28 15:13:05.546519 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:05.546484 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:13:05.546705 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:05.546484 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:13:07.080009 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:07.079948 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ckhtc" podUID="70961166-700d-40ad-867d-dcdf297e6704" May 28 15:13:10.324396 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.324358 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:13:10.324396 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.324403 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:13:10.324864 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.324526 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:13:10.326782 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.326753 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"image-registry-6dd7bd75b8-mqcfl\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:13:10.326888 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.326809 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df4dc93c-2918-4976-a0a3-800fd9078a13-metrics-tls\") pod \"dns-default-7qp66\" (UID: \"df4dc93c-2918-4976-a0a3-800fd9078a13\") " pod="openshift-dns/dns-default-7qp66" May 28 15:13:10.326888 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.326818 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7574eb4-3b1e-478d-9f61-dbf80303c7ea-cert\") pod \"ingress-canary-ggbmj\" (UID: \"f7574eb4-3b1e-478d-9f61-dbf80303c7ea\") " pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:13:10.350518 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.350493 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ljmrm\"" May 28 15:13:10.351492 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.351478 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-glc25\"" May 28 15:13:10.357957 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.357938 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:13:10.358050 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.357957 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggbmj" May 28 15:13:10.491813 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.491781 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggbmj"] May 28 15:13:10.495122 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:13:10.495091 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7574eb4_3b1e_478d_9f61_dbf80303c7ea.slice/crio-bb86deb98c674e3cc933d730eb83fdd198453f99f3966b55b6d1ffc0965bfff2 WatchSource:0}: Error finding container bb86deb98c674e3cc933d730eb83fdd198453f99f3966b55b6d1ffc0965bfff2: Status 404 returned error can't find the container with id bb86deb98c674e3cc933d730eb83fdd198453f99f3966b55b6d1ffc0965bfff2 May 28 15:13:10.519566 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.519534 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dd7bd75b8-mqcfl"] May 28 15:13:10.522179 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:13:10.522152 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d2a06f_69eb_4689_99cf_26f2aea43403.slice/crio-051f09531cf47f1a5033f78f19ef8c88ebbda42e5a773bfe34462702f2bf91eb WatchSource:0}: Error finding container 051f09531cf47f1a5033f78f19ef8c88ebbda42e5a773bfe34462702f2bf91eb: Status 404 returned error can't find the container with id 051f09531cf47f1a5033f78f19ef8c88ebbda42e5a773bfe34462702f2bf91eb May 28 15:13:10.560679 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.560651 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggbmj" event={"ID":"f7574eb4-3b1e-478d-9f61-dbf80303c7ea","Type":"ContainerStarted","Data":"bb86deb98c674e3cc933d730eb83fdd198453f99f3966b55b6d1ffc0965bfff2"} May 28 15:13:10.561605 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:10.561583 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" event={"ID":"f7d2a06f-69eb-4689-99cf-26f2aea43403","Type":"ContainerStarted","Data":"051f09531cf47f1a5033f78f19ef8c88ebbda42e5a773bfe34462702f2bf91eb"} May 28 15:13:11.565461 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:11.565421 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" event={"ID":"f7d2a06f-69eb-4689-99cf-26f2aea43403","Type":"ContainerStarted","Data":"507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776"} May 28 15:13:11.565994 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:11.565595 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:13:11.588827 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:11.588787 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" podStartSLOduration=150.58877314 podStartE2EDuration="2m30.58877314s" podCreationTimestamp="2026-05-28 15:10:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:13:11.587420732 +0000 UTC m=+164.090174894" watchObservedRunningTime="2026-05-28 15:13:11.58877314 +0000 UTC m=+164.091527300" May 28 15:13:12.278113 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.278060 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" podUID="eca9c31d-2a24-45f0-8e9a-1cb49b550e4c" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused" May 28 15:13:12.570178 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.570140 2569 generic.go:358] "Generic (PLEG): container finished" podID="eca9c31d-2a24-45f0-8e9a-1cb49b550e4c" containerID="a4739b2006f795764a959a53310bc886d365d796d13884b5efddd84260624064" exitCode=1 May 28 15:13:12.570588 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.570225 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" event={"ID":"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c","Type":"ContainerDied","Data":"a4739b2006f795764a959a53310bc886d365d796d13884b5efddd84260624064"} May 28 15:13:12.570685 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.570638 2569 scope.go:117] "RemoveContainer" containerID="a4739b2006f795764a959a53310bc886d365d796d13884b5efddd84260624064" May 28 15:13:12.571522 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.571493 2569 generic.go:358] "Generic (PLEG): container finished" podID="d53a9613-b036-4eab-a8bb-7a7713ad1211" containerID="401e4b0e326636353ced98eb8d5a75581664260ad95ede8912e3845f25c0310d" exitCode=255 May 28 15:13:12.571646 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.571569 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" event={"ID":"d53a9613-b036-4eab-a8bb-7a7713ad1211","Type":"ContainerDied","Data":"401e4b0e326636353ced98eb8d5a75581664260ad95ede8912e3845f25c0310d"} May 28 15:13:12.571900 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.571885 2569 scope.go:117] "RemoveContainer" containerID="401e4b0e326636353ced98eb8d5a75581664260ad95ede8912e3845f25c0310d" May 28 15:13:12.573067 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.572950 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggbmj" event={"ID":"f7574eb4-3b1e-478d-9f61-dbf80303c7ea","Type":"ContainerStarted","Data":"846499d17b61b871a5827291578cd5a577dc7b6090df308e91c534c43e73b856"} May 28 15:13:12.614195 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.614149 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ggbmj" podStartSLOduration=128.736969699 podStartE2EDuration="2m10.614132751s" podCreationTimestamp="2026-05-28 15:11:02 +0000 UTC" firstStartedPulling="2026-05-28 15:13:10.496914594 +0000 UTC m=+162.999668734" lastFinishedPulling="2026-05-28 15:13:12.374077644 +0000 UTC m=+164.876831786" observedRunningTime="2026-05-28 15:13:12.611075412 +0000 UTC m=+165.113829572" watchObservedRunningTime="2026-05-28 15:13:12.614132751 +0000 UTC m=+165.116886912" May 28 15:13:12.653640 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.653590 2569 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" May 28 15:13:12.670843 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:12.670813 2569 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:13:13.576998 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:13.576966 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" event={"ID":"eca9c31d-2a24-45f0-8e9a-1cb49b550e4c","Type":"ContainerStarted","Data":"1810aa659cdcf6b89ec8119b663f413b59eb290238edaa089cbad5e90a3fc507"} May 28 15:13:13.577476 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:13.577273 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:13:13.578174 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:13.578150 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-695665dd76-gg6vk" May 28 15:13:13.578541 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:13.578521 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fff6d8f4-brr5n" event={"ID":"d53a9613-b036-4eab-a8bb-7a7713ad1211","Type":"ContainerStarted","Data":"4cd51a3f9b4b6c025bd24256080146ca98799152c107a7114c21596ce8eb31a1"} May 28 15:13:15.454611 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.454536 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p7wzv"] May 28 15:13:15.457494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.457477 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.462415 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.462388 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 28 15:13:15.462561 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.462445 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 28 15:13:15.462561 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.462399 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 28 15:13:15.462705 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.462408 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tqbxg\"" May 28 15:13:15.462705 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.462388 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 28 15:13:15.474522 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.474494 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7wzv"] May 28 15:13:15.569354 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.569315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7122333f-be47-41f9-96de-24884b2bca15-data-volume\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.569354 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.569354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7122333f-be47-41f9-96de-24884b2bca15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.569571 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.569375 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7122333f-be47-41f9-96de-24884b2bca15-crio-socket\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.569571 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.569470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7122333f-be47-41f9-96de-24884b2bca15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.569571 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.569511 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7pv2\" (UniqueName: \"kubernetes.io/projected/7122333f-be47-41f9-96de-24884b2bca15-kube-api-access-r7pv2\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.670821 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.670782 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7122333f-be47-41f9-96de-24884b2bca15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.671020 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.670839 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r7pv2\" (UniqueName: \"kubernetes.io/projected/7122333f-be47-41f9-96de-24884b2bca15-kube-api-access-r7pv2\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.671020 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.670895 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7122333f-be47-41f9-96de-24884b2bca15-data-volume\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.671020 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.670917 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7122333f-be47-41f9-96de-24884b2bca15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.671020 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.670946 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7122333f-be47-41f9-96de-24884b2bca15-crio-socket\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.671020 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.671011 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7122333f-be47-41f9-96de-24884b2bca15-crio-socket\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.671283 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.671261 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7122333f-be47-41f9-96de-24884b2bca15-data-volume\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.671439 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.671423 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7122333f-be47-41f9-96de-24884b2bca15-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.673292 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.673273 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7122333f-be47-41f9-96de-24884b2bca15-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.681357 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.681332 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7pv2\" (UniqueName: \"kubernetes.io/projected/7122333f-be47-41f9-96de-24884b2bca15-kube-api-access-r7pv2\") pod \"insights-runtime-extractor-p7wzv\" (UID: \"7122333f-be47-41f9-96de-24884b2bca15\") " pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.766589 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.766489 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7wzv" May 28 15:13:15.892430 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:15.892280 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7wzv"] May 28 15:13:15.895074 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:13:15.895040 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7122333f_be47_41f9_96de_24884b2bca15.slice/crio-4ba6a032efea6d58eb5272084a38ea98a61f808d77fbd7942fbaafd0c1b0904e WatchSource:0}: Error finding container 4ba6a032efea6d58eb5272084a38ea98a61f808d77fbd7942fbaafd0c1b0904e: Status 404 returned error can't find the container with id 4ba6a032efea6d58eb5272084a38ea98a61f808d77fbd7942fbaafd0c1b0904e May 28 15:13:16.587793 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:16.587759 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7wzv" event={"ID":"7122333f-be47-41f9-96de-24884b2bca15","Type":"ContainerStarted","Data":"92089ae1db970d0e851e6c5298867eeb2bf7b82858a1a2051a1daa72e6a83e21"} May 28 15:13:16.588180 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:16.587799 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7wzv" event={"ID":"7122333f-be47-41f9-96de-24884b2bca15","Type":"ContainerStarted","Data":"4ba6a032efea6d58eb5272084a38ea98a61f808d77fbd7942fbaafd0c1b0904e"} May 28 15:13:17.591807 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:17.591768 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7wzv" event={"ID":"7122333f-be47-41f9-96de-24884b2bca15","Type":"ContainerStarted","Data":"c736b397b5e531e208b94f30291ae883ab6debe6e690c55af186135513bcb5a5"} May 28 15:13:18.071242 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:18.071216 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7qp66" May 28 15:13:18.074322 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:18.074306 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8f6nh\"" May 28 15:13:18.082181 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:18.082169 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7qp66" May 28 15:13:18.207237 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:18.207165 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7qp66"] May 28 15:13:18.210854 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:13:18.210831 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4dc93c_2918_4976_a0a3_800fd9078a13.slice/crio-0348dccde2c03136ec880d993c171fdfcc34cdaa5d3562d972180837017d6303 WatchSource:0}: Error finding container 0348dccde2c03136ec880d993c171fdfcc34cdaa5d3562d972180837017d6303: Status 404 returned error can't find the container with id 0348dccde2c03136ec880d993c171fdfcc34cdaa5d3562d972180837017d6303 May 28 15:13:18.598998 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:18.598959 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7wzv" event={"ID":"7122333f-be47-41f9-96de-24884b2bca15","Type":"ContainerStarted","Data":"a693b80f2247f46cfc92a5e7501bfee2b150aee3386f3690e9c32a35e339244d"} May 28 15:13:18.599892 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:18.599872 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qp66" event={"ID":"df4dc93c-2918-4976-a0a3-800fd9078a13","Type":"ContainerStarted","Data":"0348dccde2c03136ec880d993c171fdfcc34cdaa5d3562d972180837017d6303"} May 28 15:13:18.619822 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:18.619781 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p7wzv" podStartSLOduration=1.6531811859999999 podStartE2EDuration="3.619770951s" podCreationTimestamp="2026-05-28 15:13:15 +0000 UTC" firstStartedPulling="2026-05-28 15:13:15.946329597 +0000 UTC m=+168.449083736" lastFinishedPulling="2026-05-28 15:13:17.912919358 +0000 UTC m=+170.415673501" observedRunningTime="2026-05-28 15:13:18.619137718 +0000 UTC m=+171.121891904" watchObservedRunningTime="2026-05-28 15:13:18.619770951 +0000 UTC m=+171.122525150" May 28 15:13:20.069360 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:20.069328 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:13:20.607176 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:20.607136 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qp66" event={"ID":"df4dc93c-2918-4976-a0a3-800fd9078a13","Type":"ContainerStarted","Data":"fb1b465c47dc3b12eb8a3bc01c441301e391aa1917c41637e6dcdde72d12005d"} May 28 15:13:20.607176 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:20.607172 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qp66" event={"ID":"df4dc93c-2918-4976-a0a3-800fd9078a13","Type":"ContainerStarted","Data":"331d3a9d9473ef153ee01e2894148f4b849c18167e883a587bd0a7ab6d750f36"} May 28 15:13:20.607406 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:20.607343 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7qp66" May 28 15:13:20.624954 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:20.624888 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-7qp66" podStartSLOduration=136.732952503 podStartE2EDuration="2m18.624871713s" podCreationTimestamp="2026-05-28 15:11:02 +0000 UTC" firstStartedPulling="2026-05-28 15:13:18.212735984 +0000 UTC m=+170.715490123" lastFinishedPulling="2026-05-28 15:13:20.104655184 +0000 UTC m=+172.607409333" observedRunningTime="2026-05-28 15:13:20.624063735 +0000 UTC m=+173.126817897" watchObservedRunningTime="2026-05-28 15:13:20.624871713 +0000 UTC m=+173.127625874" May 28 15:13:26.913848 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.913814 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ls7p9"] May 28 15:13:26.917202 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.917185 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.923836 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:26.923803 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" May 28 15:13:26.923836 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.923824 2569 status_manager.go:895] "Failed to get status for pod" podUID="6ab0e46f-6489-46c5-b033-9a79dd389a4a" pod="openshift-monitoring/node-exporter-ls7p9" err="pods \"node-exporter-ls7p9\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" May 28 15:13:26.923984 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:26.923855 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" May 28 15:13:26.923984 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:26.923862 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-ggxkp\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ggxkp\"" type="*v1.Secret" May 28 15:13:26.924563 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:26.924540 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"metrics-client-ca\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" May 28 15:13:26.924600 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:26.924565 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" May 28 15:13:26.926730 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:26.926711 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" May 28 15:13:26.926788 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:26.926716 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" May 28 15:13:26.958918 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.958883 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-metrics-client-ca\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959100 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.958937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prfvl\" (UniqueName: \"kubernetes.io/projected/6ab0e46f-6489-46c5-b033-9a79dd389a4a-kube-api-access-prfvl\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959100 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.958994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-wtmp\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959100 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.959025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-sys\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959100 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.959046 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959100 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.959081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-tls\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959100 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.959097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959354 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.959144 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-root\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:26.959354 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:26.959198 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-textfile\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060269 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060230 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-prfvl\" (UniqueName: \"kubernetes.io/projected/6ab0e46f-6489-46c5-b033-9a79dd389a4a-kube-api-access-prfvl\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060471 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060280 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-wtmp\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060471 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060335 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-sys\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060471 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060374 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060471 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060410 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-tls\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060471 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060418 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-wtmp\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060471 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060427 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060792 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060473 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-root\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060792 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060486 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-sys\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060792 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060511 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-textfile\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060792 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060561 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-metrics-client-ca\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060792 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060639 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ab0e46f-6489-46c5-b033-9a79dd389a4a-root\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.060941 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.060838 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-textfile\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.966449 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.966413 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 28 15:13:27.971221 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.971198 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-metrics-client-ca\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:27.988214 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:27.988187 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 28 15:13:28.017630 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.017583 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ggxkp\"" May 28 15:13:28.060680 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:28.060652 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition May 28 15:13:28.060680 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:28.060673 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition May 28 15:13:28.060906 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:28.060653 2569 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition May 28 15:13:28.060906 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:28.060717 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-kube-rbac-proxy-config podName:6ab0e46f-6489-46c5-b033-9a79dd389a4a nodeName:}" failed. No retries permitted until 2026-05-28 15:13:28.560703004 +0000 UTC m=+181.063457148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-kube-rbac-proxy-config") pod "node-exporter-ls7p9" (UID: "6ab0e46f-6489-46c5-b033-9a79dd389a4a") : failed to sync secret cache: timed out waiting for the condition May 28 15:13:28.060906 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:28.060767 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-tls podName:6ab0e46f-6489-46c5-b033-9a79dd389a4a nodeName:}" failed. No retries permitted until 2026-05-28 15:13:28.560757313 +0000 UTC m=+181.063511452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-tls") pod "node-exporter-ls7p9" (UID: "6ab0e46f-6489-46c5-b033-9a79dd389a4a") : failed to sync secret cache: timed out waiting for the condition May 28 15:13:28.060906 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:13:28.060778 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-accelerators-collector-config podName:6ab0e46f-6489-46c5-b033-9a79dd389a4a nodeName:}" failed. No retries permitted until 2026-05-28 15:13:28.56077235 +0000 UTC m=+181.063526489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-accelerators-collector-config") pod "node-exporter-ls7p9" (UID: "6ab0e46f-6489-46c5-b033-9a79dd389a4a") : failed to sync configmap cache: timed out waiting for the condition May 28 15:13:28.122430 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.122397 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 28 15:13:28.241509 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.241427 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 28 15:13:28.297933 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.297905 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 28 15:13:28.309408 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.309382 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prfvl\" (UniqueName: \"kubernetes.io/projected/6ab0e46f-6489-46c5-b033-9a79dd389a4a-kube-api-access-prfvl\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.445611 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.445577 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 28 15:13:28.570776 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.570742 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.570968 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.570789 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-tls\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.570968 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.570808 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.571385 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.571360 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.573061 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.573042 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.573154 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.573103 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ab0e46f-6489-46c5-b033-9a79dd389a4a-node-exporter-tls\") pod \"node-exporter-ls7p9\" (UID: \"6ab0e46f-6489-46c5-b033-9a79dd389a4a\") " pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.725599 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:28.725540 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ls7p9" May 28 15:13:28.733518 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:13:28.733490 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab0e46f_6489_46c5_b033_9a79dd389a4a.slice/crio-9c413694ac49f0caa77bdc14ec05dfe2c9f5ba70be66784c8ad45bd17a0de0a8 WatchSource:0}: Error finding container 9c413694ac49f0caa77bdc14ec05dfe2c9f5ba70be66784c8ad45bd17a0de0a8: Status 404 returned error can't find the container with id 9c413694ac49f0caa77bdc14ec05dfe2c9f5ba70be66784c8ad45bd17a0de0a8 May 28 15:13:29.629235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:29.629199 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls7p9" event={"ID":"6ab0e46f-6489-46c5-b033-9a79dd389a4a","Type":"ContainerStarted","Data":"899213ed153ed8d5807623a218c7b29205c2702dec987d51e12fe9af0686ac6b"} May 28 15:13:29.629235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:29.629240 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls7p9" event={"ID":"6ab0e46f-6489-46c5-b033-9a79dd389a4a","Type":"ContainerStarted","Data":"9c413694ac49f0caa77bdc14ec05dfe2c9f5ba70be66784c8ad45bd17a0de0a8"} May 28 15:13:30.361945 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:30.361910 2569 patch_prober.go:28] interesting pod/image-registry-6dd7bd75b8-mqcfl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} May 28 15:13:30.362122 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:30.361980 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 28 15:13:30.611423 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:30.611389 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7qp66" May 28 15:13:30.633131 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:30.633055 2569 generic.go:358] "Generic (PLEG): container finished" podID="6ab0e46f-6489-46c5-b033-9a79dd389a4a" containerID="899213ed153ed8d5807623a218c7b29205c2702dec987d51e12fe9af0686ac6b" exitCode=0 May 28 15:13:30.633463 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:30.633130 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls7p9" event={"ID":"6ab0e46f-6489-46c5-b033-9a79dd389a4a","Type":"ContainerDied","Data":"899213ed153ed8d5807623a218c7b29205c2702dec987d51e12fe9af0686ac6b"} May 28 15:13:31.637805 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:31.637770 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls7p9" event={"ID":"6ab0e46f-6489-46c5-b033-9a79dd389a4a","Type":"ContainerStarted","Data":"83e42b2380013519d6819d73dea9fb4d8d60b69ecd0b4842ae93483383a2e8b1"} May 28 15:13:31.637805 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:31.637805 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ls7p9" event={"ID":"6ab0e46f-6489-46c5-b033-9a79dd389a4a","Type":"ContainerStarted","Data":"0f271faf5f57df1aed2dc57cbbf12272951c6e153d1c79a85ac414ec9996d16a"} May 28 15:13:31.658555 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:31.658505 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ls7p9" podStartSLOduration=4.83647784 podStartE2EDuration="5.658490444s" podCreationTimestamp="2026-05-28 15:13:26 +0000 UTC" firstStartedPulling="2026-05-28 15:13:28.735207086 +0000 UTC m=+181.237961225" lastFinishedPulling="2026-05-28 15:13:29.557219687 +0000 UTC m=+182.059973829" observedRunningTime="2026-05-28 15:13:31.656899894 +0000 UTC m=+184.159654054" watchObservedRunningTime="2026-05-28 15:13:31.658490444 +0000 UTC m=+184.161244604" May 28 15:13:32.577412 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:32.577385 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:13:37.388202 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:13:37.388167 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd7bd75b8-mqcfl"] May 28 15:14:02.409512 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.409475 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" containerName="registry" containerID="cri-o://507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776" gracePeriod=30 May 28 15:14:02.573826 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.573792 2569 patch_prober.go:28] interesting pod/image-registry-6dd7bd75b8-mqcfl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.9:5000/healthz\": dial tcp 10.132.0.9:5000: connect: connection refused" start-of-body= May 28 15:14:02.573979 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.573859 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.9:5000/healthz\": dial tcp 10.132.0.9:5000: connect: connection refused" May 28 15:14:02.647536 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.647511 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:14:02.647946 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.647922 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" podUID="252cc408-818d-4952-b377-04d7069c75f2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" May 28 15:14:02.723397 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.723309 2569 generic.go:358] "Generic (PLEG): container finished" podID="f7d2a06f-69eb-4689-99cf-26f2aea43403" containerID="507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776" exitCode=0 May 28 15:14:02.723397 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.723373 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" May 28 15:14:02.723580 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.723393 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" event={"ID":"f7d2a06f-69eb-4689-99cf-26f2aea43403","Type":"ContainerDied","Data":"507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776"} May 28 15:14:02.723580 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.723437 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dd7bd75b8-mqcfl" event={"ID":"f7d2a06f-69eb-4689-99cf-26f2aea43403","Type":"ContainerDied","Data":"051f09531cf47f1a5033f78f19ef8c88ebbda42e5a773bfe34462702f2bf91eb"} May 28 15:14:02.723580 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.723452 2569 scope.go:117] "RemoveContainer" containerID="507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776" May 28 15:14:02.731658 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.731642 2569 scope.go:117] "RemoveContainer" containerID="507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776" May 28 15:14:02.731931 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:14:02.731912 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776\": container with ID starting with 507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776 not found: ID does not exist" containerID="507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776" May 28 15:14:02.731975 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.731946 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776"} err="failed to get container status \"507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776\": rpc error: code = NotFound desc = could not find container \"507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776\": container with ID starting with 507f06ec907997b56e0a74f8d7267523e538250696325ac29f332e27fb61f776 not found: ID does not exist" May 28 15:14:02.732054 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732041 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kjgv\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-kube-api-access-5kjgv\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732089 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732070 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732135 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732091 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-installation-pull-secrets\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732135 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732109 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-trusted-ca\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732135 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732132 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-image-registry-private-configuration\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732280 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732162 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d2a06f-69eb-4689-99cf-26f2aea43403-ca-trust-extracted\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732280 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732187 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-bound-sa-token\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732280 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732210 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-certificates\") pod \"f7d2a06f-69eb-4689-99cf-26f2aea43403\" (UID: \"f7d2a06f-69eb-4689-99cf-26f2aea43403\") " May 28 15:14:02.732598 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732554 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 28 15:14:02.732898 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.732860 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 28 15:14:02.734557 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.734525 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:14:02.734686 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.734648 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:14:02.734782 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.734760 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:14:02.734910 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.734885 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-kube-api-access-5kjgv" (OuterVolumeSpecName: "kube-api-access-5kjgv") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "kube-api-access-5kjgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:14:02.735072 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.735051 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:14:02.740522 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.740500 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d2a06f-69eb-4689-99cf-26f2aea43403-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f7d2a06f-69eb-4689-99cf-26f2aea43403" (UID: "f7d2a06f-69eb-4689-99cf-26f2aea43403"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:14:02.833141 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833109 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-installation-pull-secrets\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:02.833141 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833136 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-trusted-ca\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:02.833141 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833148 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7d2a06f-69eb-4689-99cf-26f2aea43403-image-registry-private-configuration\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:02.833344 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833158 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7d2a06f-69eb-4689-99cf-26f2aea43403-ca-trust-extracted\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:02.833344 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833167 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-bound-sa-token\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:02.833344 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833175 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-certificates\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:02.833344 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833183 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjgv\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-kube-api-access-5kjgv\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:02.833344 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:02.833193 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7d2a06f-69eb-4689-99cf-26f2aea43403-registry-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:14:03.046644 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:03.045450 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dd7bd75b8-mqcfl"] May 28 15:14:03.050396 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:03.050369 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dd7bd75b8-mqcfl"] May 28 15:14:04.072888 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:04.072852 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" path="/var/lib/kubelet/pods/f7d2a06f-69eb-4689-99cf-26f2aea43403/volumes" May 28 15:14:12.647996 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:12.647958 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" podUID="252cc408-818d-4952-b377-04d7069c75f2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" May 28 15:14:22.648228 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:22.648193 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" podUID="252cc408-818d-4952-b377-04d7069c75f2" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" May 28 15:14:22.648637 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:22.648270 2569 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" May 28 15:14:22.648830 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:22.648811 2569 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"0e9e0adf6acbe3077cf56cd92aab01b7a5ceb5385817856b1b53df5ef86e1ecd"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" containerMessage="Container service-proxy failed liveness probe, will be restarted" May 28 15:14:22.648878 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:22.648855 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" podUID="252cc408-818d-4952-b377-04d7069c75f2" containerName="service-proxy" containerID="cri-o://0e9e0adf6acbe3077cf56cd92aab01b7a5ceb5385817856b1b53df5ef86e1ecd" gracePeriod=30 May 28 15:14:22.775216 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:22.775184 2569 generic.go:358] "Generic (PLEG): container finished" podID="252cc408-818d-4952-b377-04d7069c75f2" containerID="0e9e0adf6acbe3077cf56cd92aab01b7a5ceb5385817856b1b53df5ef86e1ecd" exitCode=2 May 28 15:14:22.775345 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:22.775248 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" event={"ID":"252cc408-818d-4952-b377-04d7069c75f2","Type":"ContainerDied","Data":"0e9e0adf6acbe3077cf56cd92aab01b7a5ceb5385817856b1b53df5ef86e1ecd"} May 28 15:14:23.779942 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:23.779908 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5f67fd5d-v6kkp" event={"ID":"252cc408-818d-4952-b377-04d7069c75f2","Type":"ContainerStarted","Data":"bbf1ed4a40022b1607814e754d67d94375384832a05de48a5e93199c9b0d7e85"} May 28 15:14:39.911493 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:39.911452 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:14:39.913815 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:39.913795 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70961166-700d-40ad-867d-dcdf297e6704-metrics-certs\") pod \"network-metrics-daemon-ckhtc\" (UID: \"70961166-700d-40ad-867d-dcdf297e6704\") " pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:14:40.172554 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:40.172465 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-49m45\"" May 28 15:14:40.180688 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:40.180665 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ckhtc" May 28 15:14:40.303459 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:40.303429 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ckhtc"] May 28 15:14:40.306532 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:14:40.306506 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70961166_700d_40ad_867d_dcdf297e6704.slice/crio-817b7243ef0eb34b2a63feb9d1e8c86b515b3a5b341e1ec5dd8c8504f0d5ef3b WatchSource:0}: Error finding container 817b7243ef0eb34b2a63feb9d1e8c86b515b3a5b341e1ec5dd8c8504f0d5ef3b: Status 404 returned error can't find the container with id 817b7243ef0eb34b2a63feb9d1e8c86b515b3a5b341e1ec5dd8c8504f0d5ef3b May 28 15:14:40.827995 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:40.827952 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckhtc" event={"ID":"70961166-700d-40ad-867d-dcdf297e6704","Type":"ContainerStarted","Data":"817b7243ef0eb34b2a63feb9d1e8c86b515b3a5b341e1ec5dd8c8504f0d5ef3b"} May 28 15:14:41.834579 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:41.834544 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckhtc" event={"ID":"70961166-700d-40ad-867d-dcdf297e6704","Type":"ContainerStarted","Data":"29e86a7cac39c426de95939fd8e4fa51221885fb5f34e1a29381eaf34b2b3731"} May 28 15:14:41.834579 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:41.834578 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ckhtc" event={"ID":"70961166-700d-40ad-867d-dcdf297e6704","Type":"ContainerStarted","Data":"6617c2d6bb53bbcd5a75a84810f4e3598726cd396529007f73d2211e0a2e556e"} May 28 15:14:41.850511 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:14:41.850466 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ckhtc" podStartSLOduration=252.902938081 podStartE2EDuration="4m13.850453781s" podCreationTimestamp="2026-05-28 15:10:28 +0000 UTC" firstStartedPulling="2026-05-28 15:14:40.308474804 +0000 UTC m=+252.811228944" lastFinishedPulling="2026-05-28 15:14:41.255990491 +0000 UTC m=+253.758744644" observedRunningTime="2026-05-28 15:14:41.849059946 +0000 UTC m=+254.351814107" watchObservedRunningTime="2026-05-28 15:14:41.850453781 +0000 UTC m=+254.353207942" May 28 15:15:27.972938 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:15:27.972911 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:15:27.972938 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:15:27.972923 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:15:27.980099 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:15:27.980076 2569 kubelet.go:1628] "Image garbage collection succeeded" May 28 15:20:27.989771 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:20:27.989746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:20:27.990834 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:20:27.990814 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:23:12.711371 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.711335 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-test-apply-gqd9j"] May 28 15:23:12.711855 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.711578 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" containerName="registry" May 28 15:23:12.711855 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.711590 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" containerName="registry" May 28 15:23:12.711855 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.711650 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f7d2a06f-69eb-4689-99cf-26f2aea43403" containerName="registry" May 28 15:23:12.714605 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.714588 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-gqd9j" May 28 15:23:12.716787 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.716765 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 28 15:23:12.717732 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.717715 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" May 28 15:23:12.717793 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.717767 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 28 15:23:12.717835 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.717799 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-vzcsg\"" May 28 15:23:12.721955 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.721927 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-gqd9j"] May 28 15:23:12.783456 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.783426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmt5\" (UniqueName: \"kubernetes.io/projected/5bda200c-7712-4154-8e51-545194e2b371-kube-api-access-jqmt5\") pod \"feast-test-apply-gqd9j\" (UID: \"5bda200c-7712-4154-8e51-545194e2b371\") " pod="test-ns-feast/feast-test-apply-gqd9j" May 28 15:23:12.884423 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.884385 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmt5\" (UniqueName: \"kubernetes.io/projected/5bda200c-7712-4154-8e51-545194e2b371-kube-api-access-jqmt5\") pod \"feast-test-apply-gqd9j\" (UID: \"5bda200c-7712-4154-8e51-545194e2b371\") " pod="test-ns-feast/feast-test-apply-gqd9j" May 28 15:23:12.892217 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:12.892191 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmt5\" (UniqueName: \"kubernetes.io/projected/5bda200c-7712-4154-8e51-545194e2b371-kube-api-access-jqmt5\") pod \"feast-test-apply-gqd9j\" (UID: \"5bda200c-7712-4154-8e51-545194e2b371\") " pod="test-ns-feast/feast-test-apply-gqd9j" May 28 15:23:13.024246 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:13.024147 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-gqd9j" May 28 15:23:13.142392 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:13.142367 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-gqd9j"] May 28 15:23:13.144854 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:23:13.144828 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bda200c_7712_4154_8e51_545194e2b371.slice/crio-8bbe2994637ff4c78a760e66c48530e497fd38d1429c87bf4c95b0952811bf78 WatchSource:0}: Error finding container 8bbe2994637ff4c78a760e66c48530e497fd38d1429c87bf4c95b0952811bf78: Status 404 returned error can't find the container with id 8bbe2994637ff4c78a760e66c48530e497fd38d1429c87bf4c95b0952811bf78 May 28 15:23:13.146416 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:13.146402 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 28 15:23:14.107238 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:14.107160 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-gqd9j" event={"ID":"5bda200c-7712-4154-8e51-545194e2b371","Type":"ContainerStarted","Data":"8bbe2994637ff4c78a760e66c48530e497fd38d1429c87bf4c95b0952811bf78"} May 28 15:23:19.122135 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:19.122045 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-gqd9j" event={"ID":"5bda200c-7712-4154-8e51-545194e2b371","Type":"ContainerStarted","Data":"433f39c325b6a58a411b12ee1d99b4a80b3f0b8085c409ca2ed83bf126443154"} May 28 15:23:25.139169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:25.139084 2569 generic.go:358] "Generic (PLEG): container finished" podID="5bda200c-7712-4154-8e51-545194e2b371" containerID="433f39c325b6a58a411b12ee1d99b4a80b3f0b8085c409ca2ed83bf126443154" exitCode=0 May 28 15:23:25.139169 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:25.139156 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-gqd9j" event={"ID":"5bda200c-7712-4154-8e51-545194e2b371","Type":"ContainerDied","Data":"433f39c325b6a58a411b12ee1d99b4a80b3f0b8085c409ca2ed83bf126443154"} May 28 15:23:26.143023 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:26.142991 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-gqd9j" event={"ID":"5bda200c-7712-4154-8e51-545194e2b371","Type":"ContainerStarted","Data":"a0283773b229098728d6afd8616e838c858c07a11cf35b1d280959921596e078"} May 28 15:23:26.157678 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:26.157632 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-test-apply-gqd9j" podStartSLOduration=8.471111178 podStartE2EDuration="14.157602772s" podCreationTimestamp="2026-05-28 15:23:12 +0000 UTC" firstStartedPulling="2026-05-28 15:23:13.146529912 +0000 UTC m=+765.649284051" lastFinishedPulling="2026-05-28 15:23:18.833021494 +0000 UTC m=+771.335775645" observedRunningTime="2026-05-28 15:23:26.157274376 +0000 UTC m=+778.660028537" watchObservedRunningTime="2026-05-28 15:23:26.157602772 +0000 UTC m=+778.660356933" May 28 15:23:34.164311 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:34.164213 2569 generic.go:358] "Generic (PLEG): container finished" podID="5bda200c-7712-4154-8e51-545194e2b371" containerID="a0283773b229098728d6afd8616e838c858c07a11cf35b1d280959921596e078" exitCode=0 May 28 15:23:34.164311 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:34.164285 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-gqd9j" event={"ID":"5bda200c-7712-4154-8e51-545194e2b371","Type":"ContainerDied","Data":"a0283773b229098728d6afd8616e838c858c07a11cf35b1d280959921596e078"} May 28 15:23:35.292273 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:35.292250 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-gqd9j" May 28 15:23:35.452307 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:35.452222 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqmt5\" (UniqueName: \"kubernetes.io/projected/5bda200c-7712-4154-8e51-545194e2b371-kube-api-access-jqmt5\") pod \"5bda200c-7712-4154-8e51-545194e2b371\" (UID: \"5bda200c-7712-4154-8e51-545194e2b371\") " May 28 15:23:35.454354 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:35.454325 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bda200c-7712-4154-8e51-545194e2b371-kube-api-access-jqmt5" (OuterVolumeSpecName: "kube-api-access-jqmt5") pod "5bda200c-7712-4154-8e51-545194e2b371" (UID: "5bda200c-7712-4154-8e51-545194e2b371"). InnerVolumeSpecName "kube-api-access-jqmt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:23:35.553122 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:35.553090 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmt5\" (UniqueName: \"kubernetes.io/projected/5bda200c-7712-4154-8e51-545194e2b371-kube-api-access-jqmt5\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:23:36.170731 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:36.170699 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-gqd9j" event={"ID":"5bda200c-7712-4154-8e51-545194e2b371","Type":"ContainerDied","Data":"8bbe2994637ff4c78a760e66c48530e497fd38d1429c87bf4c95b0952811bf78"} May 28 15:23:36.170731 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:36.170729 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bbe2994637ff4c78a760e66c48530e497fd38d1429c87bf4c95b0952811bf78" May 28 15:23:36.170731 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:36.170734 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-gqd9j" May 28 15:23:36.349931 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:36.349852 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-gqd9j_5bda200c-7712-4154-8e51-545194e2b371/feast-0/0.log" May 28 15:23:36.355567 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:23:36.355549 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-gqd9j_5bda200c-7712-4154-8e51-545194e2b371/feast-1/0.log" May 28 15:24:10.347607 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:24:10.347569 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-test-apply-gqd9j"] May 28 15:24:10.355941 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:24:10.355911 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-test-apply-gqd9j"] May 28 15:24:12.072609 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:24:12.072572 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bda200c-7712-4154-8e51-545194e2b371" path="/var/lib/kubelet/pods/5bda200c-7712-4154-8e51-545194e2b371/volumes" May 28 15:25:28.006488 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:25:28.006407 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:25:28.008362 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:25:28.008338 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:28:57.001214 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.001177 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc"] May 28 15:28:57.001679 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.001419 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bda200c-7712-4154-8e51-545194e2b371" containerName="feast-0" May 28 15:28:57.001679 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.001430 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bda200c-7712-4154-8e51-545194e2b371" containerName="feast-0" May 28 15:28:57.001679 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.001438 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bda200c-7712-4154-8e51-545194e2b371" containerName="feast-1" May 28 15:28:57.001679 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.001444 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bda200c-7712-4154-8e51-545194e2b371" containerName="feast-1" May 28 15:28:57.001679 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.001485 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="5bda200c-7712-4154-8e51-545194e2b371" containerName="feast-1" May 28 15:28:57.004734 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.004715 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.006980 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.006937 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" May 28 15:28:57.007138 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.006982 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" May 28 15:28:57.007138 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.006989 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" May 28 15:28:57.007138 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.006993 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-x25z6\"" May 28 15:28:57.007138 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.006995 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 28 15:28:57.007286 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.007271 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" May 28 15:28:57.007347 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.007335 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 28 15:28:57.013828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.013806 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc"] May 28 15:28:57.067206 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.067174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c990767e-0867-4321-9e08-290fbc5e299e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.067206 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.067204 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.067382 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.067223 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.067382 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.067252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgzn7\" (UniqueName: \"kubernetes.io/projected/c990767e-0867-4321-9e08-290fbc5e299e-kube-api-access-mgzn7\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.067382 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.067301 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.067382 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.067322 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.167686 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.167649 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mgzn7\" (UniqueName: \"kubernetes.io/projected/c990767e-0867-4321-9e08-290fbc5e299e-kube-api-access-mgzn7\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.167686 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.167693 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.167949 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.167731 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.167949 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.167781 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c990767e-0867-4321-9e08-290fbc5e299e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.167949 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.167805 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.167949 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.167833 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.167949 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:28:57.167852 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found May 28 15:28:57.167949 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:28:57.167936 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found May 28 15:28:57.168177 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:28:57.167953 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found May 28 15:28:57.168177 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:28:57.167941 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls podName:c990767e-0867-4321-9e08-290fbc5e299e nodeName:}" failed. No retries permitted until 2026-05-28 15:28:57.667921123 +0000 UTC m=+1110.170675267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-9p7sc" (UID: "c990767e-0867-4321-9e08-290fbc5e299e") : secret "feast-simple-feast-setup-ui-tls" not found May 28 15:28:57.168177 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:28:57.168008 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls podName:c990767e-0867-4321-9e08-290fbc5e299e nodeName:}" failed. No retries permitted until 2026-05-28 15:28:57.66799128 +0000 UTC m=+1110.170745427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-9p7sc" (UID: "c990767e-0867-4321-9e08-290fbc5e299e") : secret "feast-simple-feast-setup-registry-tls" not found May 28 15:28:57.168177 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:28:57.168023 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls podName:c990767e-0867-4321-9e08-290fbc5e299e nodeName:}" failed. No retries permitted until 2026-05-28 15:28:57.668014124 +0000 UTC m=+1110.170768269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls") pod "feast-simple-feast-setup-664fcfcc58-9p7sc" (UID: "c990767e-0867-4321-9e08-290fbc5e299e") : secret "feast-simple-feast-setup-online-tls" not found May 28 15:28:57.168177 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.168162 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c990767e-0867-4321-9e08-290fbc5e299e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.170104 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.170086 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.176573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.176552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgzn7\" (UniqueName: \"kubernetes.io/projected/c990767e-0867-4321-9e08-290fbc5e299e-kube-api-access-mgzn7\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.671575 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.671538 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.671785 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.671596 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.671785 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.671655 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.674032 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.674000 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.674177 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.674154 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.674248 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.674154 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-9p7sc\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:57.914402 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:57.914358 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:28:58.063372 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:58.063170 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc"] May 28 15:28:58.065579 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:28:58.065553 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc990767e_0867_4321_9e08_290fbc5e299e.slice/crio-44a1742e8495614809ee41c984cf3018a92ef2deecaad77b39f71cfa62274f01 WatchSource:0}: Error finding container 44a1742e8495614809ee41c984cf3018a92ef2deecaad77b39f71cfa62274f01: Status 404 returned error can't find the container with id 44a1742e8495614809ee41c984cf3018a92ef2deecaad77b39f71cfa62274f01 May 28 15:28:58.067109 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:58.067094 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 28 15:28:58.973221 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:28:58.973184 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerStarted","Data":"44a1742e8495614809ee41c984cf3018a92ef2deecaad77b39f71cfa62274f01"} May 28 15:29:14.015236 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:14.015196 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerStarted","Data":"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160"} May 28 15:29:18.028090 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:18.027996 2569 generic.go:358] "Generic (PLEG): container finished" podID="c990767e-0867-4321-9e08-290fbc5e299e" containerID="92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160" exitCode=0 May 28 15:29:18.028090 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:18.028039 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerDied","Data":"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160"} May 28 15:29:19.032712 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:19.032675 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerStarted","Data":"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a"} May 28 15:29:24.047280 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:24.047240 2569 generic.go:358] "Generic (PLEG): container finished" podID="c990767e-0867-4321-9e08-290fbc5e299e" containerID="ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a" exitCode=0 May 28 15:29:24.047695 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:24.047314 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerDied","Data":"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a"} May 28 15:29:25.053780 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:25.053741 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerStarted","Data":"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb"} May 28 15:29:25.054264 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:25.053791 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerStarted","Data":"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065"} May 28 15:29:25.054264 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:25.053809 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerStarted","Data":"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c"} May 28 15:29:25.054264 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:25.053822 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerStarted","Data":"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a"} May 28 15:29:25.076670 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:25.076587 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podStartSLOduration=14.02592624 podStartE2EDuration="29.076566572s" podCreationTimestamp="2026-05-28 15:28:56 +0000 UTC" firstStartedPulling="2026-05-28 15:28:58.067216328 +0000 UTC m=+1110.569970466" lastFinishedPulling="2026-05-28 15:29:13.117856645 +0000 UTC m=+1125.620610798" observedRunningTime="2026-05-28 15:29:25.075195756 +0000 UTC m=+1137.577949954" watchObservedRunningTime="2026-05-28 15:29:25.076566572 +0000 UTC m=+1137.579320736" May 28 15:29:27.915007 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.914888 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.915494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.915087 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.915494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.915204 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.915494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.915247 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.915494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.915262 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.915494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.915274 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.915494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.915286 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.915955 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.915932 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:27.917133 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.917105 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.14:8816: connect: connection refused" May 28 15:29:27.917347 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.917134 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.14:6571: connect: connection refused" May 28 15:29:27.917464 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.917133 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.14:8443: connect: connection refused" May 28 15:29:27.917464 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:27.917191 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.14:6567/health\": dial tcp 10.132.0.14:6567: connect: connection refused" May 28 15:29:28.028102 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:28.028065 2569 scope.go:117] "RemoveContainer" containerID="a0283773b229098728d6afd8616e838c858c07a11cf35b1d280959921596e078" May 28 15:29:28.038193 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:28.038169 2569 scope.go:117] "RemoveContainer" containerID="433f39c325b6a58a411b12ee1d99b4a80b3f0b8085c409ca2ed83bf126443154" May 28 15:29:30.915935 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:30.915907 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:30.916370 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:30.916012 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:30.916370 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:30.916243 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:30.916756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:30.916734 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:30.916856 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:30.916847 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:30.917073 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:30.917057 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:30.920502 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:30.920478 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:29:31.076674 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:29:31.076643 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:30:28.024030 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:28.024003 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:30:28.025730 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:28.025712 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:30:33.369603 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.369562 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc"] May 28 15:30:33.370205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.369930 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="registry" containerID="cri-o://6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a" gracePeriod=30 May 28 15:30:33.370205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.369990 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" containerID="cri-o://9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb" gracePeriod=30 May 28 15:30:33.370205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.370011 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" containerID="cri-o://91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c" gracePeriod=30 May 28 15:30:33.370205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.370161 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="offline" containerID="cri-o://cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065" gracePeriod=30 May 28 15:30:33.744134 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.744043 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg"] May 28 15:30:33.747573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.747553 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.749840 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.749818 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-55hrb\"" May 28 15:30:33.756818 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.756779 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg"] May 28 15:30:33.893785 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.893756 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.893913 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.893805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbsd\" (UniqueName: \"kubernetes.io/projected/b344cefb-b8f1-484d-a556-fc5c8477b63e-kube-api-access-rgbsd\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.893913 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.893851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b344cefb-b8f1-484d-a556-fc5c8477b63e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.893913 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.893870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.893913 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.893888 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.893913 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.893903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.994909 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.994823 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b344cefb-b8f1-484d-a556-fc5c8477b63e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.994909 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.994863 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.994909 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.994894 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.994909 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.994913 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.995207 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.994963 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.995207 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.994998 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbsd\" (UniqueName: \"kubernetes.io/projected/b344cefb-b8f1-484d-a556-fc5c8477b63e-kube-api-access-rgbsd\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.995207 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:30:33.995122 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found May 28 15:30:33.995207 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:30:33.995201 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls podName:b344cefb-b8f1-484d-a556-fc5c8477b63e nodeName:}" failed. No retries permitted until 2026-05-28 15:30:34.495179162 +0000 UTC m=+1206.997933317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-8r4cg" (UID: "b344cefb-b8f1-484d-a556-fc5c8477b63e") : secret "feast-simple-feast-setup-registry-tls" not found May 28 15:30:33.995422 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.995370 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b344cefb-b8f1-484d-a556-fc5c8477b63e-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.997477 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.997440 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.997631 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.997483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:33.997702 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:33.997676 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:34.005123 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.005093 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbsd\" (UniqueName: \"kubernetes.io/projected/b344cefb-b8f1-484d-a556-fc5c8477b63e-kube-api-access-rgbsd\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:34.235800 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.235766 2569 generic.go:358] "Generic (PLEG): container finished" podID="c990767e-0867-4321-9e08-290fbc5e299e" containerID="9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb" exitCode=0 May 28 15:30:34.235800 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.235790 2569 generic.go:358] "Generic (PLEG): container finished" podID="c990767e-0867-4321-9e08-290fbc5e299e" containerID="91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c" exitCode=0 May 28 15:30:34.236010 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.235839 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerDied","Data":"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb"} May 28 15:30:34.236010 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.235880 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerDied","Data":"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c"} May 28 15:30:34.499969 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.499929 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:34.502416 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.502390 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-8r4cg\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:34.659015 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.658978 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:34.786279 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:34.786249 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg"] May 28 15:30:34.789199 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:30:34.789168 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb344cefb_b8f1_484d_a556_fc5c8477b63e.slice/crio-f478a2104424a48a556bc0bc539f7ac5c6280a677c05f83ba34168d036b9a5da WatchSource:0}: Error finding container f478a2104424a48a556bc0bc539f7ac5c6280a677c05f83ba34168d036b9a5da: Status 404 returned error can't find the container with id f478a2104424a48a556bc0bc539f7ac5c6280a677c05f83ba34168d036b9a5da May 28 15:30:35.239699 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:35.239583 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerStarted","Data":"4781c18556ba03f13d3e72dc79c21bde27faa16df2a3956a23ad2846137bb7f7"} May 28 15:30:35.239699 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:35.239632 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerStarted","Data":"f478a2104424a48a556bc0bc539f7ac5c6280a677c05f83ba34168d036b9a5da"} May 28 15:30:40.254086 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:40.254048 2569 generic.go:358] "Generic (PLEG): container finished" podID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerID="4781c18556ba03f13d3e72dc79c21bde27faa16df2a3956a23ad2846137bb7f7" exitCode=0 May 28 15:30:40.254468 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:40.254123 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerDied","Data":"4781c18556ba03f13d3e72dc79c21bde27faa16df2a3956a23ad2846137bb7f7"} May 28 15:30:40.917763 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:40.917662 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.14:8443: connect: connection refused" May 28 15:30:41.074385 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:41.074332 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.14:6567/health\": dial tcp 10.132.0.14:6567: connect: connection refused" May 28 15:30:41.259461 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:41.259360 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerStarted","Data":"ff8f854ec6be744625979c958d9ada62a648316b04055953f1df8443568418bf"} May 28 15:30:46.274258 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:46.274222 2569 generic.go:358] "Generic (PLEG): container finished" podID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerID="ff8f854ec6be744625979c958d9ada62a648316b04055953f1df8443568418bf" exitCode=0 May 28 15:30:46.274661 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:46.274293 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerDied","Data":"ff8f854ec6be744625979c958d9ada62a648316b04055953f1df8443568418bf"} May 28 15:30:47.281691 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:47.281648 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerStarted","Data":"223d56236d1d14b11498dd55587a92a90243edaeb2f61b52698a939358011206"} May 28 15:30:47.281691 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:47.281694 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerStarted","Data":"3966d64092ca68609b7a3790a897fb644be6d7bb91e3c7410049b1f69b416cbd"} May 28 15:30:47.282232 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:47.281709 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerStarted","Data":"8914f89aa7dbdce9d0b1ac221eea15e1cf4e47e880cd1f3e3d7496d1e65d8cd1"} May 28 15:30:47.282232 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:47.281721 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerStarted","Data":"39898ac42aed9b7e6453e03e10a694fbafde3269c4fd441a829b1ec2f7f772ff"} May 28 15:30:47.305700 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:47.305640 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podStartSLOduration=14.305583692999999 podStartE2EDuration="14.305583693s" podCreationTimestamp="2026-05-28 15:30:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:30:47.302982402 +0000 UTC m=+1219.805736588" watchObservedRunningTime="2026-05-28 15:30:47.305583693 +0000 UTC m=+1219.808337855" May 28 15:30:49.659657 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.659099 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:49.659657 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.659152 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:49.659657 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.659294 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:49.660638 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.660283 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:49.661068 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.660916 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.15:6571: connect: connection refused" May 28 15:30:49.661068 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.660945 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.15:8443: connect: connection refused" May 28 15:30:49.661068 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.660985 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.15:6567/health\": dial tcp 10.132.0.15:6567: connect: connection refused" May 28 15:30:49.661068 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:49.661030 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.15:8816: connect: connection refused" May 28 15:30:50.917793 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:50.917745 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.14:8443: connect: connection refused" May 28 15:30:51.074452 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:51.074404 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.14:6567/health\": dial tcp 10.132.0.14:6567: connect: connection refused" May 28 15:30:52.659853 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:52.659824 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:52.662645 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:52.661219 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:52.662645 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:52.661257 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:52.662645 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:52.661528 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:52.665039 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:52.662934 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:52.667445 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:52.667422 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:53.300455 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:53.300392 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:53.300455 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:53.300432 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:53.300455 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:53.300445 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:53.300865 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:53.300845 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:53.301056 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:53.301040 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:30:53.303208 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:30:53.303196 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:31:00.917414 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:00.917363 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.14:8443: connect: connection refused" May 28 15:31:00.917912 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:00.917520 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:31:01.073941 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:01.073894 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.14:6567/health\": dial tcp 10.132.0.14:6567: connect: connection refused" May 28 15:31:01.074126 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:01.074036 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:31:04.016897 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.016873 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:31:04.126282 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.126255 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls\") pod \"c990767e-0867-4321-9e08-290fbc5e299e\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " May 28 15:31:04.126465 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.126312 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls\") pod \"c990767e-0867-4321-9e08-290fbc5e299e\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " May 28 15:31:04.126465 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.126343 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgzn7\" (UniqueName: \"kubernetes.io/projected/c990767e-0867-4321-9e08-290fbc5e299e-kube-api-access-mgzn7\") pod \"c990767e-0867-4321-9e08-290fbc5e299e\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " May 28 15:31:04.126465 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.126361 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-offline-tls\") pod \"c990767e-0867-4321-9e08-290fbc5e299e\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " May 28 15:31:04.126465 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.126377 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c990767e-0867-4321-9e08-290fbc5e299e-feast-data\") pod \"c990767e-0867-4321-9e08-290fbc5e299e\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " May 28 15:31:04.126465 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.126396 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls\") pod \"c990767e-0867-4321-9e08-290fbc5e299e\" (UID: \"c990767e-0867-4321-9e08-290fbc5e299e\") " May 28 15:31:04.127066 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.127016 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c990767e-0867-4321-9e08-290fbc5e299e-feast-data" (OuterVolumeSpecName: "feast-data") pod "c990767e-0867-4321-9e08-290fbc5e299e" (UID: "c990767e-0867-4321-9e08-290fbc5e299e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:31:04.128674 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.128646 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c990767e-0867-4321-9e08-290fbc5e299e" (UID: "c990767e-0867-4321-9e08-290fbc5e299e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:31:04.128674 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.128664 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c990767e-0867-4321-9e08-290fbc5e299e" (UID: "c990767e-0867-4321-9e08-290fbc5e299e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:31:04.128826 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.128735 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls" (OuterVolumeSpecName: "online-tls") pod "c990767e-0867-4321-9e08-290fbc5e299e" (UID: "c990767e-0867-4321-9e08-290fbc5e299e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:31:04.128881 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.128856 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c990767e-0867-4321-9e08-290fbc5e299e-kube-api-access-mgzn7" (OuterVolumeSpecName: "kube-api-access-mgzn7") pod "c990767e-0867-4321-9e08-290fbc5e299e" (UID: "c990767e-0867-4321-9e08-290fbc5e299e"). InnerVolumeSpecName "kube-api-access-mgzn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:31:04.128919 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.128900 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c990767e-0867-4321-9e08-290fbc5e299e" (UID: "c990767e-0867-4321-9e08-290fbc5e299e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:31:04.227011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.226929 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgzn7\" (UniqueName: \"kubernetes.io/projected/c990767e-0867-4321-9e08-290fbc5e299e-kube-api-access-mgzn7\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:31:04.227011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.226959 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-offline-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:31:04.227011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.226968 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c990767e-0867-4321-9e08-290fbc5e299e-feast-data\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:31:04.227011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.226976 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-ui-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:31:04.227011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.226984 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-registry-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:31:04.227011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.226991 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c990767e-0867-4321-9e08-290fbc5e299e-online-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:31:04.331545 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.331513 2569 generic.go:358] "Generic (PLEG): container finished" podID="c990767e-0867-4321-9e08-290fbc5e299e" containerID="cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065" exitCode=137 May 28 15:31:04.331545 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.331536 2569 generic.go:358] "Generic (PLEG): container finished" podID="c990767e-0867-4321-9e08-290fbc5e299e" containerID="6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a" exitCode=137 May 28 15:31:04.331756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.331569 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerDied","Data":"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065"} May 28 15:31:04.331756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.331581 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" May 28 15:31:04.331756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.331590 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerDied","Data":"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a"} May 28 15:31:04.331756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.331601 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc" event={"ID":"c990767e-0867-4321-9e08-290fbc5e299e","Type":"ContainerDied","Data":"44a1742e8495614809ee41c984cf3018a92ef2deecaad77b39f71cfa62274f01"} May 28 15:31:04.331756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.331614 2569 scope.go:117] "RemoveContainer" containerID="9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb" May 28 15:31:04.340436 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.340415 2569 scope.go:117] "RemoveContainer" containerID="cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065" May 28 15:31:04.347341 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.347325 2569 scope.go:117] "RemoveContainer" containerID="91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c" May 28 15:31:04.353422 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.353399 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc"] May 28 15:31:04.354657 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.354639 2569 scope.go:117] "RemoveContainer" containerID="6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a" May 28 15:31:04.359099 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.359078 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-9p7sc"] May 28 15:31:04.361456 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.361435 2569 scope.go:117] "RemoveContainer" containerID="ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a" May 28 15:31:04.368129 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.368114 2569 scope.go:117] "RemoveContainer" containerID="92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160" May 28 15:31:04.374608 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.374455 2569 scope.go:117] "RemoveContainer" containerID="9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb" May 28 15:31:04.376230 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:04.376114 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb\": container with ID starting with 9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb not found: ID does not exist" containerID="9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb" May 28 15:31:04.376230 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.376149 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb"} err="failed to get container status \"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb\": rpc error: code = NotFound desc = could not find container \"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb\": container with ID starting with 9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb not found: ID does not exist" May 28 15:31:04.376230 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.376175 2569 scope.go:117] "RemoveContainer" containerID="cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065" May 28 15:31:04.376815 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:04.376700 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065\": container with ID starting with cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065 not found: ID does not exist" containerID="cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065" May 28 15:31:04.376815 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.376731 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065"} err="failed to get container status \"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065\": rpc error: code = NotFound desc = could not find container \"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065\": container with ID starting with cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065 not found: ID does not exist" May 28 15:31:04.376815 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.376754 2569 scope.go:117] "RemoveContainer" containerID="91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c" May 28 15:31:04.378187 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:04.378163 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c\": container with ID starting with 91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c not found: ID does not exist" containerID="91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c" May 28 15:31:04.378290 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.378190 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c"} err="failed to get container status \"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c\": rpc error: code = NotFound desc = could not find container \"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c\": container with ID starting with 91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c not found: ID does not exist" May 28 15:31:04.378290 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.378211 2569 scope.go:117] "RemoveContainer" containerID="6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a" May 28 15:31:04.378933 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:04.378915 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a\": container with ID starting with 6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a not found: ID does not exist" containerID="6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a" May 28 15:31:04.378986 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.378939 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a"} err="failed to get container status \"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a\": rpc error: code = NotFound desc = could not find container \"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a\": container with ID starting with 6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a not found: ID does not exist" May 28 15:31:04.378986 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.378954 2569 scope.go:117] "RemoveContainer" containerID="ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a" May 28 15:31:04.379219 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:04.379200 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a\": container with ID starting with ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a not found: ID does not exist" containerID="ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a" May 28 15:31:04.379272 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379225 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a"} err="failed to get container status \"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a\": rpc error: code = NotFound desc = could not find container \"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a\": container with ID starting with ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a not found: ID does not exist" May 28 15:31:04.379272 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379238 2569 scope.go:117] "RemoveContainer" containerID="92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160" May 28 15:31:04.379430 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:04.379413 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160\": container with ID starting with 92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160 not found: ID does not exist" containerID="92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160" May 28 15:31:04.379510 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379440 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160"} err="failed to get container status \"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160\": rpc error: code = NotFound desc = could not find container \"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160\": container with ID starting with 92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160 not found: ID does not exist" May 28 15:31:04.379510 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379464 2569 scope.go:117] "RemoveContainer" containerID="9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb" May 28 15:31:04.379754 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379729 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb"} err="failed to get container status \"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb\": rpc error: code = NotFound desc = could not find container \"9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb\": container with ID starting with 9b2c1a499b78c923df28d4a435b43fae67202a4fb940f6869040208d6d408fbb not found: ID does not exist" May 28 15:31:04.379814 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379754 2569 scope.go:117] "RemoveContainer" containerID="cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065" May 28 15:31:04.379961 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379946 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065"} err="failed to get container status \"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065\": rpc error: code = NotFound desc = could not find container \"cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065\": container with ID starting with cd98c797af1c4ed16a68eaf0f0c392d32ea8d6c6a2d5df5dc985606e797d8065 not found: ID does not exist" May 28 15:31:04.380019 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.379961 2569 scope.go:117] "RemoveContainer" containerID="91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c" May 28 15:31:04.380142 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.380125 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c"} err="failed to get container status \"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c\": rpc error: code = NotFound desc = could not find container \"91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c\": container with ID starting with 91885110b203d9542231107c2d58c718924bd4b32a220b258204b48538a2a82c not found: ID does not exist" May 28 15:31:04.380182 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.380144 2569 scope.go:117] "RemoveContainer" containerID="6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a" May 28 15:31:04.380309 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.380295 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a"} err="failed to get container status \"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a\": rpc error: code = NotFound desc = could not find container \"6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a\": container with ID starting with 6380e6bfb0b3107788d92802cf5358f825bef58a0c7a27bd92f2a9621a0e0e6a not found: ID does not exist" May 28 15:31:04.380352 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.380309 2569 scope.go:117] "RemoveContainer" containerID="ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a" May 28 15:31:04.380523 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.380502 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a"} err="failed to get container status \"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a\": rpc error: code = NotFound desc = could not find container \"ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a\": container with ID starting with ee2ad5fa22be6c7fc331f79ec4bfbf80d3cd9536d8cc23911a61887d0966f05a not found: ID does not exist" May 28 15:31:04.380586 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.380525 2569 scope.go:117] "RemoveContainer" containerID="92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160" May 28 15:31:04.380767 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:04.380751 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160"} err="failed to get container status \"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160\": rpc error: code = NotFound desc = could not find container \"92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160\": container with ID starting with 92a6d1e3682b0692690ba63b1cb36237dd31215c24d2e647672284538754f160 not found: ID does not exist" May 28 15:31:06.076523 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:06.076480 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c990767e-0867-4321-9e08-290fbc5e299e" path="/var/lib/kubelet/pods/c990767e-0867-4321-9e08-290fbc5e299e/volumes" May 28 15:31:55.574828 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.574793 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk"] May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575033 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="offline" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575044 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="offline" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575056 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="feast-init" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575062 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="feast-init" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575068 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575072 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575082 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="feast-apply" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575087 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="feast-apply" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575094 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="registry" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575099 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="registry" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575104 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575109 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575149 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="online" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575157 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="offline" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575163 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="registry" May 28 15:31:55.575235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.575170 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c990767e-0867-4321-9e08-290fbc5e299e" containerName="ui" May 28 15:31:55.577984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.577969 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.585302 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.585284 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" May 28 15:31:55.585425 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.585353 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" May 28 15:31:55.585473 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.585423 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-xhbtx\"" May 28 15:31:55.585524 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.585476 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" May 28 15:31:55.585524 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.585498 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" May 28 15:31:55.585740 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.585724 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" May 28 15:31:55.586236 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.586221 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" May 28 15:31:55.590283 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.590263 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk"] May 28 15:31:55.682320 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.682280 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/25161031-afb5-4bc5-9adc-ea60aa29808f-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.682320 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.682324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.682533 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.682353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l88x\" (UniqueName: \"kubernetes.io/projected/25161031-afb5-4bc5-9adc-ea60aa29808f-kube-api-access-4l88x\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.682533 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.682373 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.682533 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.682389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/25161031-afb5-4bc5-9adc-ea60aa29808f-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.682533 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.682409 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.783750 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.783721 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.783927 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.783757 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/25161031-afb5-4bc5-9adc-ea60aa29808f-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.783927 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.783779 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.783927 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.783820 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/25161031-afb5-4bc5-9adc-ea60aa29808f-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.783927 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.783842 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.783927 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.783877 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4l88x\" (UniqueName: \"kubernetes.io/projected/25161031-afb5-4bc5-9adc-ea60aa29808f-kube-api-access-4l88x\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.784191 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:55.783936 2569 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found May 28 15:31:55.784191 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:55.783961 2569 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found May 28 15:31:55.784191 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:55.784037 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls podName:25161031-afb5-4bc5-9adc-ea60aa29808f nodeName:}" failed. No retries permitted until 2026-05-28 15:31:56.284016779 +0000 UTC m=+1288.786770922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls") pod "feast-simple-feast-remote-setup-5cff94746b-xvpsk" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f") : secret "feast-simple-feast-remote-setup-online-tls" not found May 28 15:31:55.784191 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:31:55.784051 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls podName:25161031-afb5-4bc5-9adc-ea60aa29808f nodeName:}" failed. No retries permitted until 2026-05-28 15:31:56.284045231 +0000 UTC m=+1288.786799370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-xvpsk" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f") : secret "feast-simple-feast-remote-setup-offline-tls" not found May 28 15:31:55.784412 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.784254 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/25161031-afb5-4bc5-9adc-ea60aa29808f-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.784520 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.784502 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/25161031-afb5-4bc5-9adc-ea60aa29808f-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.786215 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.786194 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:55.792499 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:55.792469 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l88x\" (UniqueName: \"kubernetes.io/projected/25161031-afb5-4bc5-9adc-ea60aa29808f-kube-api-access-4l88x\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:56.287590 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:56.287555 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:56.287790 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:56.287609 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:56.289997 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:56.289975 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:56.290111 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:56.290088 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-xvpsk\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:56.487572 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:56.487535 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:31:56.614054 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:56.614026 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk"] May 28 15:31:56.615531 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:31:56.615500 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25161031_afb5_4bc5_9adc_ea60aa29808f.slice/crio-f7977808e0be38ea069f5b28b63293ee80d49149a73f86834656c3f4ea4bd53e WatchSource:0}: Error finding container f7977808e0be38ea069f5b28b63293ee80d49149a73f86834656c3f4ea4bd53e: Status 404 returned error can't find the container with id f7977808e0be38ea069f5b28b63293ee80d49149a73f86834656c3f4ea4bd53e May 28 15:31:57.469686 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:57.469649 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerStarted","Data":"86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96"} May 28 15:31:57.469686 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:31:57.469687 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerStarted","Data":"f7977808e0be38ea069f5b28b63293ee80d49149a73f86834656c3f4ea4bd53e"} May 28 15:32:01.485952 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:01.485869 2569 generic.go:358] "Generic (PLEG): container finished" podID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerID="86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96" exitCode=0 May 28 15:32:01.485952 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:01.485922 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerDied","Data":"86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96"} May 28 15:32:02.491128 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:02.491084 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerStarted","Data":"744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880"} May 28 15:32:07.512477 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:07.512392 2569 generic.go:358] "Generic (PLEG): container finished" podID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerID="744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880" exitCode=0 May 28 15:32:07.512844 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:07.512467 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerDied","Data":"744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880"} May 28 15:32:08.518481 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:08.518447 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerStarted","Data":"f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203"} May 28 15:32:08.519013 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:08.518490 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerStarted","Data":"3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d"} May 28 15:32:08.519013 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:08.518502 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerStarted","Data":"13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2"} May 28 15:32:08.543439 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:08.543379 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podStartSLOduration=13.543365176 podStartE2EDuration="13.543365176s" podCreationTimestamp="2026-05-28 15:31:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:32:08.541158381 +0000 UTC m=+1301.043912595" watchObservedRunningTime="2026-05-28 15:32:08.543365176 +0000 UTC m=+1301.046119336" May 28 15:32:11.487927 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:11.487882 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:11.488386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:11.487942 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:11.488386 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:11.487958 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:11.489602 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:11.489568 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.16:8816: connect: connection refused" May 28 15:32:11.489752 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:11.489600 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.16:8443: connect: connection refused" May 28 15:32:11.489752 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:11.489612 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.16:6567/health\": dial tcp 10.132.0.16:6567: connect: connection refused" May 28 15:32:14.489296 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.489262 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.489879 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.489571 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.489997 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.489970 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.490114 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.490002 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.490234 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.490218 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.490430 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.490413 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.493882 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.493856 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.536326 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.536296 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:32:14.539237 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:32:14.539218 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:33:17.016778 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.016696 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk"] May 28 15:33:17.017341 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.017027 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" containerID="cri-o://13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2" gracePeriod=30 May 28 15:33:17.017341 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.017081 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" containerID="cri-o://f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203" gracePeriod=30 May 28 15:33:17.017341 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.017130 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="offline" containerID="cri-o://3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d" gracePeriod=30 May 28 15:33:17.163160 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.163125 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg"] May 28 15:33:17.163808 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.163561 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="registry" containerID="cri-o://39898ac42aed9b7e6453e03e10a694fbafde3269c4fd441a829b1ec2f7f772ff" gracePeriod=30 May 28 15:33:17.163808 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.163646 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" containerID="cri-o://223d56236d1d14b11498dd55587a92a90243edaeb2f61b52698a939358011206" gracePeriod=30 May 28 15:33:17.163808 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.163636 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="offline" containerID="cri-o://3966d64092ca68609b7a3790a897fb644be6d7bb91e3c7410049b1f69b416cbd" gracePeriod=30 May 28 15:33:17.163808 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.163739 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" containerID="cri-o://8914f89aa7dbdce9d0b1ac221eea15e1cf4e47e880cd1f3e3d7496d1e65d8cd1" gracePeriod=30 May 28 15:33:17.703081 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.703050 2569 generic.go:358] "Generic (PLEG): container finished" podID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerID="f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203" exitCode=0 May 28 15:33:17.703203 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:17.703120 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerDied","Data":"f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203"} May 28 15:33:18.707777 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:18.707741 2569 generic.go:358] "Generic (PLEG): container finished" podID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerID="223d56236d1d14b11498dd55587a92a90243edaeb2f61b52698a939358011206" exitCode=0 May 28 15:33:18.707777 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:18.707768 2569 generic.go:358] "Generic (PLEG): container finished" podID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerID="8914f89aa7dbdce9d0b1ac221eea15e1cf4e47e880cd1f3e3d7496d1e65d8cd1" exitCode=0 May 28 15:33:18.708219 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:18.707808 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerDied","Data":"223d56236d1d14b11498dd55587a92a90243edaeb2f61b52698a939358011206"} May 28 15:33:18.708219 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:18.707848 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerDied","Data":"8914f89aa7dbdce9d0b1ac221eea15e1cf4e47e880cd1f3e3d7496d1e65d8cd1"} May 28 15:33:18.709884 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:18.709865 2569 generic.go:358] "Generic (PLEG): container finished" podID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerID="13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2" exitCode=0 May 28 15:33:18.709986 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:18.709913 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerDied","Data":"13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2"} May 28 15:33:23.300507 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:23.300454 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.15:8443: connect: connection refused" May 28 15:33:23.300944 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:23.300454 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.15:6567/health\": dial tcp 10.132.0.15:6567: connect: connection refused" May 28 15:33:24.490569 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:24.490529 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.16:8443: connect: connection refused" May 28 15:33:24.537060 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:24.537013 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.16:6567/health\": dial tcp 10.132.0.16:6567: connect: connection refused" May 28 15:33:33.301139 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:33.301089 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.15:8443: connect: connection refused" May 28 15:33:33.301139 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:33.301112 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.15:6567/health\": dial tcp 10.132.0.15:6567: connect: connection refused" May 28 15:33:34.490256 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:34.490205 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.16:8443: connect: connection refused" May 28 15:33:34.536684 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:34.536646 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.16:6567/health\": dial tcp 10.132.0.16:6567: connect: connection refused" May 28 15:33:43.300662 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:43.300595 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.15:8443: connect: connection refused" May 28 15:33:43.300662 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:43.300612 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" probeResult="failure" output="Get \"https://10.132.0.15:6567/health\": dial tcp 10.132.0.15:6567: connect: connection refused" May 28 15:33:43.301179 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:43.300758 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:33:43.301179 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:43.300820 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:33:44.490531 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:44.490484 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.16:8443: connect: connection refused" May 28 15:33:44.490983 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:44.490643 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:33:44.536585 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:44.536538 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" probeResult="failure" output="Get \"https://10.132.0.16:6567/health\": dial tcp 10.132.0.16:6567: connect: connection refused" May 28 15:33:44.536775 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:44.536697 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:33:47.653163 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.653138 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:33:47.697017 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.696986 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls\") pod \"25161031-afb5-4bc5-9adc-ea60aa29808f\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " May 28 15:33:47.697200 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.697030 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-ui-tls\") pod \"25161031-afb5-4bc5-9adc-ea60aa29808f\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " May 28 15:33:47.697200 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.697058 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l88x\" (UniqueName: \"kubernetes.io/projected/25161031-afb5-4bc5-9adc-ea60aa29808f-kube-api-access-4l88x\") pod \"25161031-afb5-4bc5-9adc-ea60aa29808f\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " May 28 15:33:47.697200 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.697077 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/25161031-afb5-4bc5-9adc-ea60aa29808f-feast-data\") pod \"25161031-afb5-4bc5-9adc-ea60aa29808f\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " May 28 15:33:47.697200 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.697106 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls\") pod \"25161031-afb5-4bc5-9adc-ea60aa29808f\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " May 28 15:33:47.697413 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.697232 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/25161031-afb5-4bc5-9adc-ea60aa29808f-registry-tls\") pod \"25161031-afb5-4bc5-9adc-ea60aa29808f\" (UID: \"25161031-afb5-4bc5-9adc-ea60aa29808f\") " May 28 15:33:47.697601 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.697567 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25161031-afb5-4bc5-9adc-ea60aa29808f-feast-data" (OuterVolumeSpecName: "feast-data") pod "25161031-afb5-4bc5-9adc-ea60aa29808f" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:33:47.697821 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.697790 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25161031-afb5-4bc5-9adc-ea60aa29808f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "25161031-afb5-4bc5-9adc-ea60aa29808f" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 28 15:33:47.699407 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.699385 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "25161031-afb5-4bc5-9adc-ea60aa29808f" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:33:47.699507 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.699422 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls" (OuterVolumeSpecName: "online-tls") pod "25161031-afb5-4bc5-9adc-ea60aa29808f" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:33:47.699573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.699559 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25161031-afb5-4bc5-9adc-ea60aa29808f-kube-api-access-4l88x" (OuterVolumeSpecName: "kube-api-access-4l88x") pod "25161031-afb5-4bc5-9adc-ea60aa29808f" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f"). InnerVolumeSpecName "kube-api-access-4l88x". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:33:47.699648 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.699567 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "25161031-afb5-4bc5-9adc-ea60aa29808f" (UID: "25161031-afb5-4bc5-9adc-ea60aa29808f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:33:47.789529 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.789442 2569 generic.go:358] "Generic (PLEG): container finished" podID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerID="3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d" exitCode=137 May 28 15:33:47.789837 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.789570 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" May 28 15:33:47.789837 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.789569 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerDied","Data":"3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d"} May 28 15:33:47.789837 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.789612 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk" event={"ID":"25161031-afb5-4bc5-9adc-ea60aa29808f","Type":"ContainerDied","Data":"f7977808e0be38ea069f5b28b63293ee80d49149a73f86834656c3f4ea4bd53e"} May 28 15:33:47.789837 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.789655 2569 scope.go:117] "RemoveContainer" containerID="f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203" May 28 15:33:47.792803 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.792780 2569 generic.go:358] "Generic (PLEG): container finished" podID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerID="3966d64092ca68609b7a3790a897fb644be6d7bb91e3c7410049b1f69b416cbd" exitCode=137 May 28 15:33:47.792803 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.792800 2569 generic.go:358] "Generic (PLEG): container finished" podID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerID="39898ac42aed9b7e6453e03e10a694fbafde3269c4fd441a829b1ec2f7f772ff" exitCode=137 May 28 15:33:47.792995 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.792827 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerDied","Data":"3966d64092ca68609b7a3790a897fb644be6d7bb91e3c7410049b1f69b416cbd"} May 28 15:33:47.792995 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.792849 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerDied","Data":"39898ac42aed9b7e6453e03e10a694fbafde3269c4fd441a829b1ec2f7f772ff"} May 28 15:33:47.798271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.798028 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-offline-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:47.798271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.798050 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/25161031-afb5-4bc5-9adc-ea60aa29808f-registry-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:47.798271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.798063 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-online-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:47.798271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.798076 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/25161031-afb5-4bc5-9adc-ea60aa29808f-ui-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:47.798271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.798091 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l88x\" (UniqueName: \"kubernetes.io/projected/25161031-afb5-4bc5-9adc-ea60aa29808f-kube-api-access-4l88x\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:47.798271 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.798105 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/25161031-afb5-4bc5-9adc-ea60aa29808f-feast-data\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:47.798759 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.798719 2569 scope.go:117] "RemoveContainer" containerID="3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d" May 28 15:33:47.806009 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.805999 2569 scope.go:117] "RemoveContainer" containerID="13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2" May 28 15:33:47.806408 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.806393 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:33:47.810929 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.810908 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk"] May 28 15:33:47.812876 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.812856 2569 scope.go:117] "RemoveContainer" containerID="744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880" May 28 15:33:47.814684 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.814666 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-xvpsk"] May 28 15:33:47.820218 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.820202 2569 scope.go:117] "RemoveContainer" containerID="86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96" May 28 15:33:47.826462 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.826449 2569 scope.go:117] "RemoveContainer" containerID="f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203" May 28 15:33:47.826712 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:33:47.826695 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203\": container with ID starting with f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203 not found: ID does not exist" containerID="f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203" May 28 15:33:47.826781 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.826720 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203"} err="failed to get container status \"f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203\": rpc error: code = NotFound desc = could not find container \"f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203\": container with ID starting with f68f2eabd05aaa8ba1f5d34a8ae380c995b72695e957908613ca10ee59dca203 not found: ID does not exist" May 28 15:33:47.826781 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.826744 2569 scope.go:117] "RemoveContainer" containerID="3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d" May 28 15:33:47.826995 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:33:47.826978 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d\": container with ID starting with 3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d not found: ID does not exist" containerID="3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d" May 28 15:33:47.827039 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.827002 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d"} err="failed to get container status \"3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d\": rpc error: code = NotFound desc = could not find container \"3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d\": container with ID starting with 3b0070b7f5422aa87d3cf1e3b64ce5cbe9f6cdbccc4757ac5bfa392608e6c11d not found: ID does not exist" May 28 15:33:47.827039 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.827018 2569 scope.go:117] "RemoveContainer" containerID="13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2" May 28 15:33:47.827245 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:33:47.827228 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2\": container with ID starting with 13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2 not found: ID does not exist" containerID="13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2" May 28 15:33:47.827284 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.827250 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2"} err="failed to get container status \"13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2\": rpc error: code = NotFound desc = could not find container \"13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2\": container with ID starting with 13819329db1352e9d70597832cb4a953a7c9f68e054060935469572d8538fbb2 not found: ID does not exist" May 28 15:33:47.827284 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.827263 2569 scope.go:117] "RemoveContainer" containerID="744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880" May 28 15:33:47.827465 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:33:47.827451 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880\": container with ID starting with 744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880 not found: ID does not exist" containerID="744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880" May 28 15:33:47.827504 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.827469 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880"} err="failed to get container status \"744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880\": rpc error: code = NotFound desc = could not find container \"744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880\": container with ID starting with 744ca914b616fa15a4f99a4911f2c08d12e011b7d3c2762c05ccd0f2acab8880 not found: ID does not exist" May 28 15:33:47.827504 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.827487 2569 scope.go:117] "RemoveContainer" containerID="86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96" May 28 15:33:47.827765 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:33:47.827748 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96\": container with ID starting with 86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96 not found: ID does not exist" containerID="86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96" May 28 15:33:47.827823 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.827771 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96"} err="failed to get container status \"86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96\": rpc error: code = NotFound desc = could not find container \"86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96\": container with ID starting with 86c0278cf6a843d6ab63a0f98ffbbcc3ac255dee0c72e41ddce7aa3fc70dba96 not found: ID does not exist" May 28 15:33:47.899050 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.899011 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-offline-tls\") pod \"b344cefb-b8f1-484d-a556-fc5c8477b63e\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " May 28 15:33:47.899050 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.899053 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-online-tls\") pod \"b344cefb-b8f1-484d-a556-fc5c8477b63e\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " May 28 15:33:47.899281 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.899079 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-ui-tls\") pod \"b344cefb-b8f1-484d-a556-fc5c8477b63e\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " May 28 15:33:47.899281 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.899266 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgbsd\" (UniqueName: \"kubernetes.io/projected/b344cefb-b8f1-484d-a556-fc5c8477b63e-kube-api-access-rgbsd\") pod \"b344cefb-b8f1-484d-a556-fc5c8477b63e\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " May 28 15:33:47.899398 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.899307 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls\") pod \"b344cefb-b8f1-484d-a556-fc5c8477b63e\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " May 28 15:33:47.899398 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.899344 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b344cefb-b8f1-484d-a556-fc5c8477b63e-feast-data\") pod \"b344cefb-b8f1-484d-a556-fc5c8477b63e\" (UID: \"b344cefb-b8f1-484d-a556-fc5c8477b63e\") " May 28 15:33:47.900005 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.899974 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b344cefb-b8f1-484d-a556-fc5c8477b63e-feast-data" (OuterVolumeSpecName: "feast-data") pod "b344cefb-b8f1-484d-a556-fc5c8477b63e" (UID: "b344cefb-b8f1-484d-a556-fc5c8477b63e"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:33:47.901238 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.901212 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b344cefb-b8f1-484d-a556-fc5c8477b63e" (UID: "b344cefb-b8f1-484d-a556-fc5c8477b63e"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:33:47.901347 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.901267 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b344cefb-b8f1-484d-a556-fc5c8477b63e" (UID: "b344cefb-b8f1-484d-a556-fc5c8477b63e"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:33:47.901347 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.901294 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b344cefb-b8f1-484d-a556-fc5c8477b63e" (UID: "b344cefb-b8f1-484d-a556-fc5c8477b63e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:33:47.901509 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.901487 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-online-tls" (OuterVolumeSpecName: "online-tls") pod "b344cefb-b8f1-484d-a556-fc5c8477b63e" (UID: "b344cefb-b8f1-484d-a556-fc5c8477b63e"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:33:47.901568 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:47.901505 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b344cefb-b8f1-484d-a556-fc5c8477b63e-kube-api-access-rgbsd" (OuterVolumeSpecName: "kube-api-access-rgbsd") pod "b344cefb-b8f1-484d-a556-fc5c8477b63e" (UID: "b344cefb-b8f1-484d-a556-fc5c8477b63e"). InnerVolumeSpecName "kube-api-access-rgbsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:33:48.000808 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.000777 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-offline-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:48.000808 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.000804 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-online-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:48.000808 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.000813 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-ui-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:48.001000 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.000821 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbsd\" (UniqueName: \"kubernetes.io/projected/b344cefb-b8f1-484d-a556-fc5c8477b63e-kube-api-access-rgbsd\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:48.001000 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.000832 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b344cefb-b8f1-484d-a556-fc5c8477b63e-registry-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:48.001000 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.000842 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b344cefb-b8f1-484d-a556-fc5c8477b63e-feast-data\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:33:48.071986 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.071912 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" path="/var/lib/kubelet/pods/25161031-afb5-4bc5-9adc-ea60aa29808f/volumes" May 28 15:33:48.797584 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.797549 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" event={"ID":"b344cefb-b8f1-484d-a556-fc5c8477b63e","Type":"ContainerDied","Data":"f478a2104424a48a556bc0bc539f7ac5c6280a677c05f83ba34168d036b9a5da"} May 28 15:33:48.798073 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.797601 2569 scope.go:117] "RemoveContainer" containerID="223d56236d1d14b11498dd55587a92a90243edaeb2f61b52698a939358011206" May 28 15:33:48.798073 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.797567 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg" May 28 15:33:48.805605 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.805588 2569 scope.go:117] "RemoveContainer" containerID="3966d64092ca68609b7a3790a897fb644be6d7bb91e3c7410049b1f69b416cbd" May 28 15:33:48.812272 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.812253 2569 scope.go:117] "RemoveContainer" containerID="8914f89aa7dbdce9d0b1ac221eea15e1cf4e47e880cd1f3e3d7496d1e65d8cd1" May 28 15:33:48.814833 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.814736 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg"] May 28 15:33:48.819121 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.819104 2569 scope.go:117] "RemoveContainer" containerID="39898ac42aed9b7e6453e03e10a694fbafde3269c4fd441a829b1ec2f7f772ff" May 28 15:33:48.821544 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.821519 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-8r4cg"] May 28 15:33:48.825857 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.825838 2569 scope.go:117] "RemoveContainer" containerID="ff8f854ec6be744625979c958d9ada62a648316b04055953f1df8443568418bf" May 28 15:33:48.832145 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:48.832130 2569 scope.go:117] "RemoveContainer" containerID="4781c18556ba03f13d3e72dc79c21bde27faa16df2a3956a23ad2846137bb7f7" May 28 15:33:50.072674 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:33:50.072636 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" path="/var/lib/kubelet/pods/b344cefb-b8f1-484d-a556-fc5c8477b63e/volumes" May 28 15:34:27.988140 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988054 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn"] May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988439 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988459 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988470 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="feast-init" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988480 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="feast-init" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988494 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="offline" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988527 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="offline" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988543 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="offline" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988552 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="offline" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988560 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988565 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988574 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="feast-apply" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988579 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="feast-apply" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988585 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="registry" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988590 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="registry" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988599 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="feast-apply" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988604 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="feast-apply" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988612 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988634 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988644 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988649 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988654 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="feast-init" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988659 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="feast-init" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988709 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="offline" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988718 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="ui" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988724 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="registry" May 28 15:34:27.988711 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988730 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="online" May 28 15:34:27.989916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988736 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b344cefb-b8f1-484d-a556-fc5c8477b63e" containerName="ui" May 28 15:34:27.989916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988742 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="offline" May 28 15:34:27.989916 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.988749 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="25161031-afb5-4bc5-9adc-ea60aa29808f" containerName="online" May 28 15:34:27.991795 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.991775 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:27.994088 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.994065 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-gbvs2\"" May 28 15:34:27.994205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.994099 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" May 28 15:34:27.994205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.994111 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" May 28 15:34:27.994205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.994158 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 28 15:34:27.994205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.994169 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" May 28 15:34:27.995073 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.995058 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" May 28 15:34:27.995121 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:27.995084 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 28 15:34:28.004399 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.004378 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn"] May 28 15:34:28.073836 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.073809 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52p74\" (UniqueName: \"kubernetes.io/projected/741586db-e562-464e-879a-31e37e47df0b-kube-api-access-52p74\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.073984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.073866 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.073984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.073894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.073984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.073922 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/741586db-e562-464e-879a-31e37e47df0b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.073984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.073940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.073984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.073957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.174882 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.174851 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52p74\" (UniqueName: \"kubernetes.io/projected/741586db-e562-464e-879a-31e37e47df0b-kube-api-access-52p74\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.175062 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.174934 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.175062 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.174963 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.175062 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.174990 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/741586db-e562-464e-879a-31e37e47df0b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.175062 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.175018 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.175062 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.175046 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.175298 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:34:28.175060 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found May 28 15:34:28.175298 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:34:28.175134 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls podName:741586db-e562-464e-879a-31e37e47df0b nodeName:}" failed. No retries permitted until 2026-05-28 15:34:28.675112653 +0000 UTC m=+1441.177866793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-q4bkn" (UID: "741586db-e562-464e-879a-31e37e47df0b") : secret "feast-simple-feast-setup-ui-tls" not found May 28 15:34:28.175298 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:34:28.175139 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found May 28 15:34:28.175298 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:34:28.175208 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls podName:741586db-e562-464e-879a-31e37e47df0b nodeName:}" failed. No retries permitted until 2026-05-28 15:34:28.675171679 +0000 UTC m=+1441.177925832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls") pod "feast-simple-feast-setup-664fcfcc58-q4bkn" (UID: "741586db-e562-464e-879a-31e37e47df0b") : secret "feast-simple-feast-setup-registry-tls" not found May 28 15:34:28.175491 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.175470 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/741586db-e562-464e-879a-31e37e47df0b-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.177510 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.177483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.177607 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.177560 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.183664 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.183643 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52p74\" (UniqueName: \"kubernetes.io/projected/741586db-e562-464e-879a-31e37e47df0b-kube-api-access-52p74\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.678226 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.678187 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.678409 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.678234 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.680595 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.680576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.680743 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.680721 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-q4bkn\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:28.900582 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:28.900552 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:29.025759 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:29.025690 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn"] May 28 15:34:29.028043 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:34:29.028016 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod741586db_e562_464e_879a_31e37e47df0b.slice/crio-53877d0e9fa10dcac2a19a01aae1e7331047c962b65e5092f54d3ed0d67d8343 WatchSource:0}: Error finding container 53877d0e9fa10dcac2a19a01aae1e7331047c962b65e5092f54d3ed0d67d8343: Status 404 returned error can't find the container with id 53877d0e9fa10dcac2a19a01aae1e7331047c962b65e5092f54d3ed0d67d8343 May 28 15:34:29.029650 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:29.029631 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 28 15:34:29.901345 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:29.901304 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerStarted","Data":"7b33f8d8562da94f8f4d8534c050182d15107e648d7d3f71a3f92b76e3505e84"} May 28 15:34:29.901487 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:29.901352 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerStarted","Data":"53877d0e9fa10dcac2a19a01aae1e7331047c962b65e5092f54d3ed0d67d8343"} May 28 15:34:33.916212 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:33.916119 2569 generic.go:358] "Generic (PLEG): container finished" podID="741586db-e562-464e-879a-31e37e47df0b" containerID="7b33f8d8562da94f8f4d8534c050182d15107e648d7d3f71a3f92b76e3505e84" exitCode=0 May 28 15:34:33.916212 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:33.916171 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerDied","Data":"7b33f8d8562da94f8f4d8534c050182d15107e648d7d3f71a3f92b76e3505e84"} May 28 15:34:34.920411 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:34.920377 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerStarted","Data":"ff9ac9661e68e5c052aaacd90ed0c7ad4536d5b24b6856b6349c9a87b62bcbef"} May 28 15:34:39.935111 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:39.935076 2569 generic.go:358] "Generic (PLEG): container finished" podID="741586db-e562-464e-879a-31e37e47df0b" containerID="ff9ac9661e68e5c052aaacd90ed0c7ad4536d5b24b6856b6349c9a87b62bcbef" exitCode=0 May 28 15:34:39.935452 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:39.935129 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerDied","Data":"ff9ac9661e68e5c052aaacd90ed0c7ad4536d5b24b6856b6349c9a87b62bcbef"} May 28 15:34:40.941506 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:40.941464 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerStarted","Data":"b15f2fa08f0459dcde38348ca5c8be127516871d07c5d343117e7a5cd6c50ff4"} May 28 15:34:40.942115 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:40.941522 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerStarted","Data":"a8a4876535bf83bf9a126ea4d3d88d94abd4de28191af730e9eec55256f89c3b"} May 28 15:34:40.942115 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:40.941539 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerStarted","Data":"975c88f8aa972caa76c06338ead3e3c3fb54e5ca50c337a4cac26f4b0aa88824"} May 28 15:34:40.942115 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:40.941555 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerStarted","Data":"09f5c5e5da762851040aa1c6e095b498fd1a6f458ba028b30324453cc136eba3"} May 28 15:34:40.965029 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:40.964960 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podStartSLOduration=13.964939198 podStartE2EDuration="13.964939198s" podCreationTimestamp="2026-05-28 15:34:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:34:40.962193368 +0000 UTC m=+1453.464947529" watchObservedRunningTime="2026-05-28 15:34:40.964939198 +0000 UTC m=+1453.467693360" May 28 15:34:43.901855 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.900929 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:43.901855 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.900977 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:43.901855 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.900992 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:43.901855 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.901004 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:43.902747 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.902707 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.17:8443: connect: connection refused" May 28 15:34:43.902747 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.902718 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.17:6571: connect: connection refused" May 28 15:34:43.902925 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.902750 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.17:8816: connect: connection refused" May 28 15:34:43.902925 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:43.902817 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.17:6567/health\": dial tcp 10.132.0.17:6567: connect: connection refused" May 28 15:34:46.902552 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.902520 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.902985 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.902634 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.902985 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.902887 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.903280 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.903255 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.903424 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.903406 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.903521 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.903428 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.903521 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.903486 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.903594 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.903531 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.903754 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.903736 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.907103 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.907086 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.959849 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.959813 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:34:46.962653 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:34:46.962636 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:35:28.040428 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:28.040397 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:35:28.042365 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:28.042341 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:35:47.962138 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:47.962048 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn"] May 28 15:35:47.962639 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:47.962535 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="registry" containerID="cri-o://09f5c5e5da762851040aa1c6e095b498fd1a6f458ba028b30324453cc136eba3" gracePeriod=30 May 28 15:35:47.962761 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:47.962737 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" containerID="cri-o://b15f2fa08f0459dcde38348ca5c8be127516871d07c5d343117e7a5cd6c50ff4" gracePeriod=30 May 28 15:35:47.962931 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:47.962807 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" containerID="cri-o://975c88f8aa972caa76c06338ead3e3c3fb54e5ca50c337a4cac26f4b0aa88824" gracePeriod=30 May 28 15:35:47.963029 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:47.962827 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="offline" containerID="cri-o://a8a4876535bf83bf9a126ea4d3d88d94abd4de28191af730e9eec55256f89c3b" gracePeriod=30 May 28 15:35:48.351961 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.351920 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk"] May 28 15:35:48.356279 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.356254 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.358682 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.358654 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-qwlkl\"" May 28 15:35:48.365567 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.365534 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk"] May 28 15:35:48.454470 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.454432 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.454650 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.454484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.454650 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.454516 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.454650 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.454637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c2136386-df60-48f2-aff3-39a67285cb36-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.454798 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.454677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.454798 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.454751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjxq\" (UniqueName: \"kubernetes.io/projected/c2136386-df60-48f2-aff3-39a67285cb36-kube-api-access-wtjxq\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.555870 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.555834 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c2136386-df60-48f2-aff3-39a67285cb36-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.556052 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.555879 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.556052 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.555924 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjxq\" (UniqueName: \"kubernetes.io/projected/c2136386-df60-48f2-aff3-39a67285cb36-kube-api-access-wtjxq\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.556052 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.555962 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.556052 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.555989 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.556258 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.556209 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.556308 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.556292 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c2136386-df60-48f2-aff3-39a67285cb36-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.558445 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.558414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.558693 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.558670 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.558804 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.558790 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.558898 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.558881 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.562578 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.562557 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjxq\" (UniqueName: \"kubernetes.io/projected/c2136386-df60-48f2-aff3-39a67285cb36-kube-api-access-wtjxq\") pod \"feast-simple-feast-setup-664fcfcc58-f5lnk\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.668125 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.668095 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:35:48.792202 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:48.792169 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk"] May 28 15:35:48.795189 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:35:48.795161 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2136386_df60_48f2_aff3_39a67285cb36.slice/crio-beb0d15750dec21e70defefbfb88a6680b048a4386914d40b4d9aca6292f79f9 WatchSource:0}: Error finding container beb0d15750dec21e70defefbfb88a6680b048a4386914d40b4d9aca6292f79f9: Status 404 returned error can't find the container with id beb0d15750dec21e70defefbfb88a6680b048a4386914d40b4d9aca6292f79f9 May 28 15:35:49.119961 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:49.119921 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerStarted","Data":"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4"} May 28 15:35:49.119961 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:49.119965 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerStarted","Data":"beb0d15750dec21e70defefbfb88a6680b048a4386914d40b4d9aca6292f79f9"} May 28 15:35:49.122705 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:49.122673 2569 generic.go:358] "Generic (PLEG): container finished" podID="741586db-e562-464e-879a-31e37e47df0b" containerID="b15f2fa08f0459dcde38348ca5c8be127516871d07c5d343117e7a5cd6c50ff4" exitCode=0 May 28 15:35:49.122705 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:49.122705 2569 generic.go:358] "Generic (PLEG): container finished" podID="741586db-e562-464e-879a-31e37e47df0b" containerID="975c88f8aa972caa76c06338ead3e3c3fb54e5ca50c337a4cac26f4b0aa88824" exitCode=0 May 28 15:35:49.122848 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:49.122736 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerDied","Data":"b15f2fa08f0459dcde38348ca5c8be127516871d07c5d343117e7a5cd6c50ff4"} May 28 15:35:49.122848 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:49.122764 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerDied","Data":"975c88f8aa972caa76c06338ead3e3c3fb54e5ca50c337a4cac26f4b0aa88824"} May 28 15:35:54.137443 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:54.137407 2569 generic.go:358] "Generic (PLEG): container finished" podID="c2136386-df60-48f2-aff3-39a67285cb36" containerID="7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4" exitCode=0 May 28 15:35:54.137844 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:54.137446 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerDied","Data":"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4"} May 28 15:35:55.141837 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:55.141802 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerStarted","Data":"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001"} May 28 15:35:56.904292 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:56.904242 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.17:8443: connect: connection refused" May 28 15:35:56.960811 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:35:56.960759 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.17:6567/health\": dial tcp 10.132.0.17:6567: connect: connection refused" May 28 15:36:00.156040 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:00.156007 2569 generic.go:358] "Generic (PLEG): container finished" podID="c2136386-df60-48f2-aff3-39a67285cb36" containerID="4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001" exitCode=0 May 28 15:36:00.156448 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:00.156081 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerDied","Data":"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001"} May 28 15:36:01.163698 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:01.163654 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerStarted","Data":"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9"} May 28 15:36:01.164191 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:01.163706 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerStarted","Data":"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a"} May 28 15:36:01.164191 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:01.163721 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerStarted","Data":"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a"} May 28 15:36:01.164191 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:01.163733 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerStarted","Data":"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231"} May 28 15:36:01.188894 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:01.188810 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podStartSLOduration=13.18878915 podStartE2EDuration="13.18878915s" podCreationTimestamp="2026-05-28 15:35:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:36:01.187317385 +0000 UTC m=+1533.690071584" watchObservedRunningTime="2026-05-28 15:36:01.18878915 +0000 UTC m=+1533.691543315" May 28 15:36:03.669387 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.669353 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:03.669942 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.669925 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:03.670043 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.670034 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:03.670515 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.670494 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:03.671655 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.671497 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.18:8443: connect: connection refused" May 28 15:36:03.671655 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.671531 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.18:6571: connect: connection refused" May 28 15:36:03.671655 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.671541 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.18:8816: connect: connection refused" May 28 15:36:03.671655 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:03.671599 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" probeResult="failure" output="Get \"https://10.132.0.18:6567/health\": dial tcp 10.132.0.18:6567: connect: connection refused" May 28 15:36:06.669710 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.669682 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670156 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.669771 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670156 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.670068 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670427 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.670407 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670513 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.670438 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670513 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.670451 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670607 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.670567 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670706 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.670680 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.670868 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.670856 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.674267 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.674248 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:06.903602 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.903554 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.17:8443: connect: connection refused" May 28 15:36:06.961069 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:06.960972 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.17:6567/health\": dial tcp 10.132.0.17:6567: connect: connection refused" May 28 15:36:07.190950 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:07.190923 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:07.193848 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:07.193832 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:36:16.904120 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:16.904069 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.17:8443: connect: connection refused" May 28 15:36:16.904521 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:16.904214 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:36:16.960468 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:16.960425 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" probeResult="failure" output="Get \"https://10.132.0.17:6567/health\": dial tcp 10.132.0.17:6567: connect: connection refused" May 28 15:36:16.960676 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:16.960556 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:36:18.223016 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.222928 2569 generic.go:358] "Generic (PLEG): container finished" podID="741586db-e562-464e-879a-31e37e47df0b" containerID="a8a4876535bf83bf9a126ea4d3d88d94abd4de28191af730e9eec55256f89c3b" exitCode=137 May 28 15:36:18.223016 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.222954 2569 generic.go:358] "Generic (PLEG): container finished" podID="741586db-e562-464e-879a-31e37e47df0b" containerID="09f5c5e5da762851040aa1c6e095b498fd1a6f458ba028b30324453cc136eba3" exitCode=137 May 28 15:36:18.223016 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.222983 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerDied","Data":"a8a4876535bf83bf9a126ea4d3d88d94abd4de28191af730e9eec55256f89c3b"} May 28 15:36:18.223420 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.223021 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerDied","Data":"09f5c5e5da762851040aa1c6e095b498fd1a6f458ba028b30324453cc136eba3"} May 28 15:36:18.603511 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.603479 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:36:18.691696 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.691664 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52p74\" (UniqueName: \"kubernetes.io/projected/741586db-e562-464e-879a-31e37e47df0b-kube-api-access-52p74\") pod \"741586db-e562-464e-879a-31e37e47df0b\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " May 28 15:36:18.691696 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.691707 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-online-tls\") pod \"741586db-e562-464e-879a-31e37e47df0b\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " May 28 15:36:18.691919 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.691734 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls\") pod \"741586db-e562-464e-879a-31e37e47df0b\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " May 28 15:36:18.691919 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.691782 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/741586db-e562-464e-879a-31e37e47df0b-feast-data\") pod \"741586db-e562-464e-879a-31e37e47df0b\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " May 28 15:36:18.691919 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.691807 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls\") pod \"741586db-e562-464e-879a-31e37e47df0b\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " May 28 15:36:18.691919 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.691830 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-offline-tls\") pod \"741586db-e562-464e-879a-31e37e47df0b\" (UID: \"741586db-e562-464e-879a-31e37e47df0b\") " May 28 15:36:18.692345 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.692309 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741586db-e562-464e-879a-31e37e47df0b-feast-data" (OuterVolumeSpecName: "feast-data") pod "741586db-e562-464e-879a-31e37e47df0b" (UID: "741586db-e562-464e-879a-31e37e47df0b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:36:18.694001 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.693970 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "741586db-e562-464e-879a-31e37e47df0b" (UID: "741586db-e562-464e-879a-31e37e47df0b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:36:18.694380 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.694349 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-online-tls" (OuterVolumeSpecName: "online-tls") pod "741586db-e562-464e-879a-31e37e47df0b" (UID: "741586db-e562-464e-879a-31e37e47df0b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:36:18.694459 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.694373 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "741586db-e562-464e-879a-31e37e47df0b" (UID: "741586db-e562-464e-879a-31e37e47df0b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:36:18.694459 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.694424 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "741586db-e562-464e-879a-31e37e47df0b" (UID: "741586db-e562-464e-879a-31e37e47df0b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:36:18.694459 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.694438 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741586db-e562-464e-879a-31e37e47df0b-kube-api-access-52p74" (OuterVolumeSpecName: "kube-api-access-52p74") pod "741586db-e562-464e-879a-31e37e47df0b" (UID: "741586db-e562-464e-879a-31e37e47df0b"). InnerVolumeSpecName "kube-api-access-52p74". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:36:18.792583 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.792540 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/741586db-e562-464e-879a-31e37e47df0b-feast-data\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:36:18.792583 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.792582 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-registry-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:36:18.792583 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.792595 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-offline-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:36:18.792847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.792608 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52p74\" (UniqueName: \"kubernetes.io/projected/741586db-e562-464e-879a-31e37e47df0b-kube-api-access-52p74\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:36:18.792847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.792644 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-online-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:36:18.792847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:18.792656 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/741586db-e562-464e-879a-31e37e47df0b-ui-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:36:19.228333 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.228252 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" event={"ID":"741586db-e562-464e-879a-31e37e47df0b","Type":"ContainerDied","Data":"53877d0e9fa10dcac2a19a01aae1e7331047c962b65e5092f54d3ed0d67d8343"} May 28 15:36:19.228333 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.228299 2569 scope.go:117] "RemoveContainer" containerID="b15f2fa08f0459dcde38348ca5c8be127516871d07c5d343117e7a5cd6c50ff4" May 28 15:36:19.228333 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.228314 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn" May 28 15:36:19.242716 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.242632 2569 scope.go:117] "RemoveContainer" containerID="a8a4876535bf83bf9a126ea4d3d88d94abd4de28191af730e9eec55256f89c3b" May 28 15:36:19.249536 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.249518 2569 scope.go:117] "RemoveContainer" containerID="975c88f8aa972caa76c06338ead3e3c3fb54e5ca50c337a4cac26f4b0aa88824" May 28 15:36:19.251812 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.251790 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn"] May 28 15:36:19.254988 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.254967 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-q4bkn"] May 28 15:36:19.256974 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.256954 2569 scope.go:117] "RemoveContainer" containerID="09f5c5e5da762851040aa1c6e095b498fd1a6f458ba028b30324453cc136eba3" May 28 15:36:19.263383 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.263366 2569 scope.go:117] "RemoveContainer" containerID="ff9ac9661e68e5c052aaacd90ed0c7ad4536d5b24b6856b6349c9a87b62bcbef" May 28 15:36:19.269597 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:19.269579 2569 scope.go:117] "RemoveContainer" containerID="7b33f8d8562da94f8f4d8534c050182d15107e648d7d3f71a3f92b76e3505e84" May 28 15:36:20.073286 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:36:20.073255 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741586db-e562-464e-879a-31e37e47df0b" path="/var/lib/kubelet/pods/741586db-e562-464e-879a-31e37e47df0b/volumes" May 28 15:37:09.802806 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.802770 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf"] May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803039 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="feast-apply" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803051 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="feast-apply" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803063 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="offline" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803068 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="offline" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803074 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="registry" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803080 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="registry" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803090 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="feast-init" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803095 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="feast-init" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803102 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803109 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803119 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803124 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803166 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="registry" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803174 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="online" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803181 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="offline" May 28 15:37:09.803265 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.803193 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="741586db-e562-464e-879a-31e37e47df0b" containerName="ui" May 28 15:37:09.806338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.806322 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.808724 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.808700 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" May 28 15:37:09.808847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.808708 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" May 28 15:37:09.808847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.808808 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" May 28 15:37:09.808962 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.808944 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" May 28 15:37:09.809055 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.809038 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-snkf8\"" May 28 15:37:09.809116 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.809064 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" May 28 15:37:09.809170 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.809136 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" May 28 15:37:09.818568 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.818540 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf"] May 28 15:37:09.840790 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.840761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.840790 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.840797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.840985 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.840833 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.840985 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.840915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.840985 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.840947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.840985 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.840968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vt2x\" (UniqueName: \"kubernetes.io/projected/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-kube-api-access-7vt2x\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.941483 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.941440 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.941690 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.941496 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.941690 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.941519 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.941690 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.941537 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7vt2x\" (UniqueName: \"kubernetes.io/projected/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-kube-api-access-7vt2x\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.941690 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:37:09.941654 2569 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found May 28 15:37:09.941915 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.941709 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.941915 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:37:09.941721 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls podName:1ab628f2-fbcb-476d-afb8-8a5d4f705b70 nodeName:}" failed. No retries permitted until 2026-05-28 15:37:10.441699878 +0000 UTC m=+1602.944454021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-77rcf" (UID: "1ab628f2-fbcb-476d-afb8-8a5d4f705b70") : secret "feast-simple-feast-remote-setup-offline-tls" not found May 28 15:37:09.941915 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.941776 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.942080 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.941925 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.942414 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.942393 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.943989 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.943963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.944078 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.943996 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:09.949853 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:09.949832 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vt2x\" (UniqueName: \"kubernetes.io/projected/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-kube-api-access-7vt2x\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:10.445240 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:10.445202 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:10.447669 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:10.447646 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-77rcf\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:10.720431 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:10.720336 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:10.851578 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:10.851554 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf"] May 28 15:37:10.853465 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:37:10.853439 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ab628f2_fbcb_476d_afb8_8a5d4f705b70.slice/crio-291f00adda2639a8496b9d245ae7e6a3ead5740b8d86f5df7e76ebb3ee64e9d9 WatchSource:0}: Error finding container 291f00adda2639a8496b9d245ae7e6a3ead5740b8d86f5df7e76ebb3ee64e9d9: Status 404 returned error can't find the container with id 291f00adda2639a8496b9d245ae7e6a3ead5740b8d86f5df7e76ebb3ee64e9d9 May 28 15:37:11.364011 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:11.363977 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerStarted","Data":"4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df"} May 28 15:37:11.364188 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:11.364019 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerStarted","Data":"291f00adda2639a8496b9d245ae7e6a3ead5740b8d86f5df7e76ebb3ee64e9d9"} May 28 15:37:16.379871 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:16.379834 2569 generic.go:358] "Generic (PLEG): container finished" podID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerID="4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df" exitCode=0 May 28 15:37:16.380242 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:16.379889 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerDied","Data":"4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df"} May 28 15:37:17.384345 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:17.384305 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerStarted","Data":"f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655"} May 28 15:37:22.399941 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:22.399844 2569 generic.go:358] "Generic (PLEG): container finished" podID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerID="f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655" exitCode=0 May 28 15:37:22.399941 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:22.399884 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerDied","Data":"f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655"} May 28 15:37:23.405817 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:23.405781 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerStarted","Data":"6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16"} May 28 15:37:23.405817 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:23.405823 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerStarted","Data":"a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606"} May 28 15:37:23.406362 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:23.405833 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerStarted","Data":"daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61"} May 28 15:37:23.429043 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:23.428981 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podStartSLOduration=14.428965586 podStartE2EDuration="14.428965586s" podCreationTimestamp="2026-05-28 15:37:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:37:23.426979621 +0000 UTC m=+1615.929733784" watchObservedRunningTime="2026-05-28 15:37:23.428965586 +0000 UTC m=+1615.931719747" May 28 15:37:25.720703 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:25.720665 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:25.721231 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:25.721208 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:25.721231 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:25.721234 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:25.722330 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:25.722299 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.19:8816: connect: connection refused" May 28 15:37:25.722460 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:25.722338 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" May 28 15:37:25.722460 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:25.722348 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" May 28 15:37:28.722266 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:28.722236 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:28.722674 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:28.722441 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:28.722838 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:28.722802 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:28.722838 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:28.722831 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:28.723170 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:28.723149 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:28.723223 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:28.723214 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:28.726102 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:28.726085 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:29.424984 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:29.424944 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:37:29.427909 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:37:29.427887 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:38:30.788121 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.788086 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf"] May 28 15:38:30.788658 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.788523 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" containerID="cri-o://daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61" gracePeriod=30 May 28 15:38:30.788658 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.788590 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" containerID="cri-o://6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16" gracePeriod=30 May 28 15:38:30.788779 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.788593 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="offline" containerID="cri-o://a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606" gracePeriod=30 May 28 15:38:30.934972 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.934936 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk"] May 28 15:38:30.935529 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.935356 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="registry" containerID="cri-o://40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231" gracePeriod=30 May 28 15:38:30.935529 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.935398 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" containerID="cri-o://8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9" gracePeriod=30 May 28 15:38:30.935529 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.935412 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="offline" containerID="cri-o://8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a" gracePeriod=30 May 28 15:38:30.935830 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:30.935546 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" containerID="cri-o://afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a" gracePeriod=30 May 28 15:38:31.593274 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:31.593240 2569 generic.go:358] "Generic (PLEG): container finished" podID="c2136386-df60-48f2-aff3-39a67285cb36" containerID="8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9" exitCode=0 May 28 15:38:31.593440 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:31.593292 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerDied","Data":"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9"} May 28 15:38:31.595546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:31.595522 2569 generic.go:358] "Generic (PLEG): container finished" podID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerID="6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16" exitCode=0 May 28 15:38:31.595546 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:31.595541 2569 generic.go:358] "Generic (PLEG): container finished" podID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerID="daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61" exitCode=0 May 28 15:38:31.595720 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:31.595572 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerDied","Data":"6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16"} May 28 15:38:31.595720 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:31.595591 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerDied","Data":"daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61"} May 28 15:38:32.600485 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:32.600452 2569 generic.go:358] "Generic (PLEG): container finished" podID="c2136386-df60-48f2-aff3-39a67285cb36" containerID="afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a" exitCode=0 May 28 15:38:32.600862 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:32.600525 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerDied","Data":"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a"} May 28 15:38:36.670843 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:36.670794 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.18:8443: connect: connection refused" May 28 15:38:37.191448 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:37.191412 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" probeResult="failure" output="Get \"https://10.132.0.18:6567/health\": dial tcp 10.132.0.18:6567: connect: connection refused" May 28 15:38:38.723877 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:38.723837 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" May 28 15:38:39.425247 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:39.425210 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" May 28 15:38:46.670762 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:46.670661 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.18:8443: connect: connection refused" May 28 15:38:47.191978 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:47.191936 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" probeResult="failure" output="Get \"https://10.132.0.18:6567/health\": dial tcp 10.132.0.18:6567: connect: connection refused" May 28 15:38:48.723086 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:48.723032 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" May 28 15:38:49.425154 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:49.425111 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" May 28 15:38:56.670692 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:56.670646 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.18:8443: connect: connection refused" May 28 15:38:56.671129 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:56.670793 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:38:57.191327 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:57.191291 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" probeResult="failure" output="Get \"https://10.132.0.18:6567/health\": dial tcp 10.132.0.18:6567: connect: connection refused" May 28 15:38:57.191494 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:57.191404 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:38:58.723847 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:58.723799 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.19:8443: connect: connection refused" May 28 15:38:58.724311 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:58.723961 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:38:59.425587 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:59.425545 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" probeResult="failure" output="Get \"https://10.132.0.19:6567/health\": dial tcp 10.132.0.19:6567: connect: connection refused" May 28 15:38:59.425783 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:38:59.425693 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:39:01.037513 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.037490 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:39:01.142813 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.142722 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-online-tls\") pod \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " May 28 15:39:01.142813 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.142773 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-registry-tls\") pod \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " May 28 15:39:01.142813 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.142804 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls\") pod \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " May 28 15:39:01.143088 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.142825 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vt2x\" (UniqueName: \"kubernetes.io/projected/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-kube-api-access-7vt2x\") pod \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " May 28 15:39:01.143088 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.142854 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-feast-data\") pod \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " May 28 15:39:01.143088 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.142888 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-ui-tls\") pod \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\" (UID: \"1ab628f2-fbcb-476d-afb8-8a5d4f705b70\") " May 28 15:39:01.143287 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.143250 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1ab628f2-fbcb-476d-afb8-8a5d4f705b70" (UID: "1ab628f2-fbcb-476d-afb8-8a5d4f705b70"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 28 15:39:01.143413 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.143382 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-feast-data" (OuterVolumeSpecName: "feast-data") pod "1ab628f2-fbcb-476d-afb8-8a5d4f705b70" (UID: "1ab628f2-fbcb-476d-afb8-8a5d4f705b70"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:39:01.145106 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.145077 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-online-tls" (OuterVolumeSpecName: "online-tls") pod "1ab628f2-fbcb-476d-afb8-8a5d4f705b70" (UID: "1ab628f2-fbcb-476d-afb8-8a5d4f705b70"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:39:01.145221 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.145103 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1ab628f2-fbcb-476d-afb8-8a5d4f705b70" (UID: "1ab628f2-fbcb-476d-afb8-8a5d4f705b70"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:39:01.145221 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.145158 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1ab628f2-fbcb-476d-afb8-8a5d4f705b70" (UID: "1ab628f2-fbcb-476d-afb8-8a5d4f705b70"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:39:01.145221 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.145177 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-kube-api-access-7vt2x" (OuterVolumeSpecName: "kube-api-access-7vt2x") pod "1ab628f2-fbcb-476d-afb8-8a5d4f705b70" (UID: "1ab628f2-fbcb-476d-afb8-8a5d4f705b70"). InnerVolumeSpecName "kube-api-access-7vt2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:39:01.243805 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.243764 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-online-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.243805 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.243799 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-registry-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.243805 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.243811 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-offline-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.244037 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.243823 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vt2x\" (UniqueName: \"kubernetes.io/projected/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-kube-api-access-7vt2x\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.244037 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.243837 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-feast-data\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.244037 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.243849 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1ab628f2-fbcb-476d-afb8-8a5d4f705b70-ui-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.574235 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.574211 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:39:01.651381 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.651336 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-offline-tls\") pod \"c2136386-df60-48f2-aff3-39a67285cb36\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " May 28 15:39:01.651681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.651397 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c2136386-df60-48f2-aff3-39a67285cb36-feast-data\") pod \"c2136386-df60-48f2-aff3-39a67285cb36\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " May 28 15:39:01.651681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.651422 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-online-tls\") pod \"c2136386-df60-48f2-aff3-39a67285cb36\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " May 28 15:39:01.651681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.651458 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-ui-tls\") pod \"c2136386-df60-48f2-aff3-39a67285cb36\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " May 28 15:39:01.651681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.651491 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtjxq\" (UniqueName: \"kubernetes.io/projected/c2136386-df60-48f2-aff3-39a67285cb36-kube-api-access-wtjxq\") pod \"c2136386-df60-48f2-aff3-39a67285cb36\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " May 28 15:39:01.651681 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.651557 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-registry-tls\") pod \"c2136386-df60-48f2-aff3-39a67285cb36\" (UID: \"c2136386-df60-48f2-aff3-39a67285cb36\") " May 28 15:39:01.652006 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.651977 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2136386-df60-48f2-aff3-39a67285cb36-feast-data" (OuterVolumeSpecName: "feast-data") pod "c2136386-df60-48f2-aff3-39a67285cb36" (UID: "c2136386-df60-48f2-aff3-39a67285cb36"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:39:01.653534 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.653508 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-online-tls" (OuterVolumeSpecName: "online-tls") pod "c2136386-df60-48f2-aff3-39a67285cb36" (UID: "c2136386-df60-48f2-aff3-39a67285cb36"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:39:01.653654 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.653572 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c2136386-df60-48f2-aff3-39a67285cb36" (UID: "c2136386-df60-48f2-aff3-39a67285cb36"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:39:01.653654 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.653586 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c2136386-df60-48f2-aff3-39a67285cb36" (UID: "c2136386-df60-48f2-aff3-39a67285cb36"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:39:01.653867 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.653851 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c2136386-df60-48f2-aff3-39a67285cb36" (UID: "c2136386-df60-48f2-aff3-39a67285cb36"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 28 15:39:01.653920 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.653863 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2136386-df60-48f2-aff3-39a67285cb36-kube-api-access-wtjxq" (OuterVolumeSpecName: "kube-api-access-wtjxq") pod "c2136386-df60-48f2-aff3-39a67285cb36" (UID: "c2136386-df60-48f2-aff3-39a67285cb36"). InnerVolumeSpecName "kube-api-access-wtjxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:39:01.681336 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.681306 2569 generic.go:358] "Generic (PLEG): container finished" podID="c2136386-df60-48f2-aff3-39a67285cb36" containerID="8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a" exitCode=137 May 28 15:39:01.681336 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.681332 2569 generic.go:358] "Generic (PLEG): container finished" podID="c2136386-df60-48f2-aff3-39a67285cb36" containerID="40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231" exitCode=137 May 28 15:39:01.681554 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.681373 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" May 28 15:39:01.681554 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.681392 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerDied","Data":"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a"} May 28 15:39:01.681554 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.681434 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerDied","Data":"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231"} May 28 15:39:01.681554 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.681450 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk" event={"ID":"c2136386-df60-48f2-aff3-39a67285cb36","Type":"ContainerDied","Data":"beb0d15750dec21e70defefbfb88a6680b048a4386914d40b4d9aca6292f79f9"} May 28 15:39:01.681554 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.681471 2569 scope.go:117] "RemoveContainer" containerID="8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9" May 28 15:39:01.683727 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.683702 2569 generic.go:358] "Generic (PLEG): container finished" podID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerID="a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606" exitCode=137 May 28 15:39:01.683850 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.683784 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerDied","Data":"a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606"} May 28 15:39:01.683850 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.683810 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" event={"ID":"1ab628f2-fbcb-476d-afb8-8a5d4f705b70","Type":"ContainerDied","Data":"291f00adda2639a8496b9d245ae7e6a3ead5740b8d86f5df7e76ebb3ee64e9d9"} May 28 15:39:01.683850 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.683810 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf" May 28 15:39:01.689771 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.689754 2569 scope.go:117] "RemoveContainer" containerID="8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a" May 28 15:39:01.697385 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.697367 2569 scope.go:117] "RemoveContainer" containerID="afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a" May 28 15:39:01.704123 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.704075 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk"] May 28 15:39:01.705172 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.705157 2569 scope.go:117] "RemoveContainer" containerID="40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231" May 28 15:39:01.706865 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.706846 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-f5lnk"] May 28 15:39:01.712214 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.712198 2569 scope.go:117] "RemoveContainer" containerID="4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001" May 28 15:39:01.718211 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.718192 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf"] May 28 15:39:01.719006 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.718995 2569 scope.go:117] "RemoveContainer" containerID="7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4" May 28 15:39:01.722752 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.722730 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-77rcf"] May 28 15:39:01.725902 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.725887 2569 scope.go:117] "RemoveContainer" containerID="8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9" May 28 15:39:01.726136 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.726118 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9\": container with ID starting with 8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9 not found: ID does not exist" containerID="8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9" May 28 15:39:01.726194 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726144 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9"} err="failed to get container status \"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9\": rpc error: code = NotFound desc = could not find container \"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9\": container with ID starting with 8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9 not found: ID does not exist" May 28 15:39:01.726194 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726162 2569 scope.go:117] "RemoveContainer" containerID="8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a" May 28 15:39:01.726412 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.726396 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a\": container with ID starting with 8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a not found: ID does not exist" containerID="8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a" May 28 15:39:01.726448 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726418 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a"} err="failed to get container status \"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a\": rpc error: code = NotFound desc = could not find container \"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a\": container with ID starting with 8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a not found: ID does not exist" May 28 15:39:01.726448 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726434 2569 scope.go:117] "RemoveContainer" containerID="afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a" May 28 15:39:01.726707 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.726680 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a\": container with ID starting with afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a not found: ID does not exist" containerID="afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a" May 28 15:39:01.726756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726710 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a"} err="failed to get container status \"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a\": rpc error: code = NotFound desc = could not find container \"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a\": container with ID starting with afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a not found: ID does not exist" May 28 15:39:01.726756 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726722 2569 scope.go:117] "RemoveContainer" containerID="40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231" May 28 15:39:01.726962 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.726945 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231\": container with ID starting with 40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231 not found: ID does not exist" containerID="40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231" May 28 15:39:01.727014 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726967 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231"} err="failed to get container status \"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231\": rpc error: code = NotFound desc = could not find container \"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231\": container with ID starting with 40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231 not found: ID does not exist" May 28 15:39:01.727014 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.726981 2569 scope.go:117] "RemoveContainer" containerID="4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001" May 28 15:39:01.727191 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.727176 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001\": container with ID starting with 4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001 not found: ID does not exist" containerID="4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001" May 28 15:39:01.727231 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727193 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001"} err="failed to get container status \"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001\": rpc error: code = NotFound desc = could not find container \"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001\": container with ID starting with 4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001 not found: ID does not exist" May 28 15:39:01.727231 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727206 2569 scope.go:117] "RemoveContainer" containerID="7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4" May 28 15:39:01.727393 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.727373 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4\": container with ID starting with 7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4 not found: ID does not exist" containerID="7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4" May 28 15:39:01.727454 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727403 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4"} err="failed to get container status \"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4\": rpc error: code = NotFound desc = could not find container \"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4\": container with ID starting with 7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4 not found: ID does not exist" May 28 15:39:01.727454 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727425 2569 scope.go:117] "RemoveContainer" containerID="8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9" May 28 15:39:01.727612 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727598 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9"} err="failed to get container status \"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9\": rpc error: code = NotFound desc = could not find container \"8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9\": container with ID starting with 8443cb264a783bb19ea4955d54de6a50ed1f46bf0d3a4dd7fbdfd68ea970f6e9 not found: ID does not exist" May 28 15:39:01.727612 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727612 2569 scope.go:117] "RemoveContainer" containerID="8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a" May 28 15:39:01.727838 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727824 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a"} err="failed to get container status \"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a\": rpc error: code = NotFound desc = could not find container \"8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a\": container with ID starting with 8a97417d6c4c6ee6141f563fb87ae38c137d82dea7d6267ac50cbe238ca77a8a not found: ID does not exist" May 28 15:39:01.727888 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727838 2569 scope.go:117] "RemoveContainer" containerID="afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a" May 28 15:39:01.727995 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727979 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a"} err="failed to get container status \"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a\": rpc error: code = NotFound desc = could not find container \"afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a\": container with ID starting with afcd38772f830bd9a9921b85c11e3d6ea65b47ba7a7701873c62b851e3c7d38a not found: ID does not exist" May 28 15:39:01.728041 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.727995 2569 scope.go:117] "RemoveContainer" containerID="40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231" May 28 15:39:01.728177 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.728163 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231"} err="failed to get container status \"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231\": rpc error: code = NotFound desc = could not find container \"40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231\": container with ID starting with 40e4d34f4692b4ee2e25e751136a7c918bb7c911c9f30381bdb3a46b4f954231 not found: ID does not exist" May 28 15:39:01.728222 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.728177 2569 scope.go:117] "RemoveContainer" containerID="4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001" May 28 15:39:01.728346 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.728333 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001"} err="failed to get container status \"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001\": rpc error: code = NotFound desc = could not find container \"4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001\": container with ID starting with 4a4b11f39bec41bec12620f26be514730eae5414d948ba45f16bc09617bfc001 not found: ID does not exist" May 28 15:39:01.728388 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.728346 2569 scope.go:117] "RemoveContainer" containerID="7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4" May 28 15:39:01.728514 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.728500 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4"} err="failed to get container status \"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4\": rpc error: code = NotFound desc = could not find container \"7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4\": container with ID starting with 7d350e2aba9f88c747010b792eb213912abeb51536d7f521030734c1a537dcd4 not found: ID does not exist" May 28 15:39:01.728560 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.728514 2569 scope.go:117] "RemoveContainer" containerID="6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16" May 28 15:39:01.735176 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.735162 2569 scope.go:117] "RemoveContainer" containerID="a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606" May 28 15:39:01.743607 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.743588 2569 scope.go:117] "RemoveContainer" containerID="daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61" May 28 15:39:01.750507 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.750480 2569 scope.go:117] "RemoveContainer" containerID="f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655" May 28 15:39:01.752503 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.752488 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-registry-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.752573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.752506 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-offline-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.752573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.752515 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c2136386-df60-48f2-aff3-39a67285cb36-feast-data\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.752573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.752524 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-online-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.752573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.752532 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c2136386-df60-48f2-aff3-39a67285cb36-ui-tls\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.752573 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.752541 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjxq\" (UniqueName: \"kubernetes.io/projected/c2136386-df60-48f2-aff3-39a67285cb36-kube-api-access-wtjxq\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:01.756730 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.756715 2569 scope.go:117] "RemoveContainer" containerID="4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df" May 28 15:39:01.762874 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.762857 2569 scope.go:117] "RemoveContainer" containerID="6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16" May 28 15:39:01.763146 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.763127 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16\": container with ID starting with 6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16 not found: ID does not exist" containerID="6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16" May 28 15:39:01.763223 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763159 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16"} err="failed to get container status \"6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16\": rpc error: code = NotFound desc = could not find container \"6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16\": container with ID starting with 6af23ccd3e094c0f57b0293eed608d0cc60b6e22b9bc70c3c499c52b750f0d16 not found: ID does not exist" May 28 15:39:01.763223 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763184 2569 scope.go:117] "RemoveContainer" containerID="a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606" May 28 15:39:01.763421 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.763404 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606\": container with ID starting with a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606 not found: ID does not exist" containerID="a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606" May 28 15:39:01.763460 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763427 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606"} err="failed to get container status \"a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606\": rpc error: code = NotFound desc = could not find container \"a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606\": container with ID starting with a3dcfc0693235c47f8516cc60df2149f2e783dd16b735148e364bd8b02485606 not found: ID does not exist" May 28 15:39:01.763460 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763443 2569 scope.go:117] "RemoveContainer" containerID="daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61" May 28 15:39:01.763688 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.763671 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61\": container with ID starting with daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61 not found: ID does not exist" containerID="daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61" May 28 15:39:01.763741 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763693 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61"} err="failed to get container status \"daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61\": rpc error: code = NotFound desc = could not find container \"daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61\": container with ID starting with daeac4045adfa7038d8ec85af5732302ca79374b4d284fcfb384a56475941d61 not found: ID does not exist" May 28 15:39:01.763741 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763707 2569 scope.go:117] "RemoveContainer" containerID="f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655" May 28 15:39:01.763930 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.763913 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655\": container with ID starting with f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655 not found: ID does not exist" containerID="f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655" May 28 15:39:01.763992 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763943 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655"} err="failed to get container status \"f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655\": rpc error: code = NotFound desc = could not find container \"f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655\": container with ID starting with f43f296669dedf106db18cd7e88dc237bf85afd184fb6d07309edc7655235655 not found: ID does not exist" May 28 15:39:01.763992 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.763963 2569 scope.go:117] "RemoveContainer" containerID="4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df" May 28 15:39:01.764191 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:01.764175 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df\": container with ID starting with 4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df not found: ID does not exist" containerID="4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df" May 28 15:39:01.764230 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:01.764195 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df"} err="failed to get container status \"4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df\": rpc error: code = NotFound desc = could not find container \"4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df\": container with ID starting with 4376458bfcd695d76476a5d3d89e18553d270aeadf2f491908821eeee46178df not found: ID does not exist" May 28 15:39:02.072448 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:02.072418 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" path="/var/lib/kubelet/pods/1ab628f2-fbcb-476d-afb8-8a5d4f705b70/volumes" May 28 15:39:02.072966 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:02.072952 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2136386-df60-48f2-aff3-39a67285cb36" path="/var/lib/kubelet/pods/c2136386-df60-48f2-aff3-39a67285cb36/volumes" May 28 15:39:19.943884 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.943842 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7n7p/must-gather-fd2kp"] May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944057 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="feast-apply" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944067 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="feast-apply" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944077 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944082 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944092 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="feast-init" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944098 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="feast-init" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944105 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="feast-apply" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944111 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="feast-apply" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944117 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944123 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944132 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944138 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944148 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="registry" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944153 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="registry" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944159 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="offline" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944163 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="offline" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944170 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944174 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944180 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="feast-init" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944186 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="feast-init" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944192 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="offline" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944196 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="offline" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944229 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="online" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944234 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="ui" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944243 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="offline" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944249 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="ui" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944257 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="online" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944264 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2136386-df60-48f2-aff3-39a67285cb36" containerName="registry" May 28 15:39:19.944338 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.944270 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ab628f2-fbcb-476d-afb8-8a5d4f705b70" containerName="offline" May 28 15:39:19.948848 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.948829 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:19.951133 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.951111 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j7n7p\"/\"default-dockercfg-sn7zb\"" May 28 15:39:19.951252 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.951114 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7n7p\"/\"kube-root-ca.crt\"" May 28 15:39:19.951252 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.951162 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7n7p\"/\"openshift-service-ca.crt\"" May 28 15:39:19.954564 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:19.954532 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7n7p/must-gather-fd2kp"] May 28 15:39:20.078205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.078170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-must-gather-output\") pod \"must-gather-fd2kp\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:20.078205 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.078209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gj8\" (UniqueName: \"kubernetes.io/projected/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-kube-api-access-w2gj8\") pod \"must-gather-fd2kp\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:20.179390 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.179353 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-must-gather-output\") pod \"must-gather-fd2kp\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:20.179390 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.179392 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gj8\" (UniqueName: \"kubernetes.io/projected/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-kube-api-access-w2gj8\") pod \"must-gather-fd2kp\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:20.179718 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.179686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-must-gather-output\") pod \"must-gather-fd2kp\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:20.187611 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.187592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gj8\" (UniqueName: \"kubernetes.io/projected/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-kube-api-access-w2gj8\") pod \"must-gather-fd2kp\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:20.257896 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.257803 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:20.378947 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.378768 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7n7p/must-gather-fd2kp"] May 28 15:39:20.381385 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:39:20.381356 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5c14ef_a4ec_4d0a_8509_3fc2faa5af72.slice/crio-debac6d19bc10b98cc2d35f8b20347bba96a82ee38e43afc56fa9666148a63fc WatchSource:0}: Error finding container debac6d19bc10b98cc2d35f8b20347bba96a82ee38e43afc56fa9666148a63fc: Status 404 returned error can't find the container with id debac6d19bc10b98cc2d35f8b20347bba96a82ee38e43afc56fa9666148a63fc May 28 15:39:20.731403 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:20.731360 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" event={"ID":"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72","Type":"ContainerStarted","Data":"debac6d19bc10b98cc2d35f8b20347bba96a82ee38e43afc56fa9666148a63fc"} May 28 15:39:26.752084 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:26.751536 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" event={"ID":"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72","Type":"ContainerStarted","Data":"64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de"} May 28 15:39:26.752084 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:26.751579 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" event={"ID":"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72","Type":"ContainerStarted","Data":"a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573"} May 28 15:39:26.765978 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:26.765922 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" podStartSLOduration=1.741726417 podStartE2EDuration="7.765908504s" podCreationTimestamp="2026-05-28 15:39:19 +0000 UTC" firstStartedPulling="2026-05-28 15:39:20.383007998 +0000 UTC m=+1732.885762137" lastFinishedPulling="2026-05-28 15:39:26.407190085 +0000 UTC m=+1738.909944224" observedRunningTime="2026-05-28 15:39:26.764869791 +0000 UTC m=+1739.267623954" watchObservedRunningTime="2026-05-28 15:39:26.765908504 +0000 UTC m=+1739.268662664" May 28 15:39:36.778667 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:36.778609 2569 generic.go:358] "Generic (PLEG): container finished" podID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerID="a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573" exitCode=0 May 28 15:39:36.779081 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:36.778681 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" event={"ID":"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72","Type":"ContainerDied","Data":"a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573"} May 28 15:39:36.779081 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:36.778998 2569 scope.go:117] "RemoveContainer" containerID="a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573" May 28 15:39:37.001882 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:37.001851 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-j7n7p_must-gather-fd2kp_4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72/gather/0.log" May 28 15:39:40.005047 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:40.005013 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bdr2w_6b36d81d-a1fb-41b3-bf1f-52cf9e95b2c7/global-pull-secret-syncer/0.log" May 28 15:39:40.185899 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:40.185866 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bg5rp_8fb0dbb2-e7bc-4fdf-9582-b5709495834a/konnectivity-agent/0.log" May 28 15:39:40.250744 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:40.250717 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-98.ec2.internal_a61332edee42088905ed1b7a0d6356e8/haproxy/0.log" May 28 15:39:42.338536 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.338502 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-j7n7p/must-gather-fd2kp"] May 28 15:39:42.338948 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.338728 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerName="copy" containerID="cri-o://64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de" gracePeriod=2 May 28 15:39:42.340911 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.340876 2569 status_manager.go:895] "Failed to get status for pod" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" err="pods \"must-gather-fd2kp\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j7n7p\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" May 28 15:39:42.341528 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.341498 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-j7n7p/must-gather-fd2kp"] May 28 15:39:42.559188 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.559167 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-j7n7p_must-gather-fd2kp_4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72/copy/0.log" May 28 15:39:42.559520 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.559506 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:42.561530 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.561510 2569 status_manager.go:895] "Failed to get status for pod" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" err="pods \"must-gather-fd2kp\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j7n7p\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" May 28 15:39:42.647857 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.647768 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2gj8\" (UniqueName: \"kubernetes.io/projected/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-kube-api-access-w2gj8\") pod \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " May 28 15:39:42.647857 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.647824 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-must-gather-output\") pod \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\" (UID: \"4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72\") " May 28 15:39:42.648375 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.648349 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" (UID: "4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 28 15:39:42.650270 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.650246 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-kube-api-access-w2gj8" (OuterVolumeSpecName: "kube-api-access-w2gj8") pod "4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" (UID: "4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72"). InnerVolumeSpecName "kube-api-access-w2gj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 28 15:39:42.748838 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.748800 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2gj8\" (UniqueName: \"kubernetes.io/projected/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-kube-api-access-w2gj8\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:42.748838 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.748830 2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72-must-gather-output\") on node \"ip-10-0-130-98.ec2.internal\" DevicePath \"\"" May 28 15:39:42.795377 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.795351 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-j7n7p_must-gather-fd2kp_4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72/copy/0.log" May 28 15:39:42.795694 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.795668 2569 generic.go:358] "Generic (PLEG): container finished" podID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerID="64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de" exitCode=143 May 28 15:39:42.795765 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.795720 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" May 28 15:39:42.795827 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.795725 2569 scope.go:117] "RemoveContainer" containerID="64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de" May 28 15:39:42.797801 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.797767 2569 status_manager.go:895] "Failed to get status for pod" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" err="pods \"must-gather-fd2kp\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j7n7p\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" May 28 15:39:42.802804 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.802787 2569 scope.go:117] "RemoveContainer" containerID="a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573" May 28 15:39:42.805610 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.805587 2569 status_manager.go:895] "Failed to get status for pod" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" pod="openshift-must-gather-j7n7p/must-gather-fd2kp" err="pods \"must-gather-fd2kp\" is forbidden: User \"system:node:ip-10-0-130-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-j7n7p\": no relationship found between node 'ip-10-0-130-98.ec2.internal' and this object" May 28 15:39:42.812905 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.812886 2569 scope.go:117] "RemoveContainer" containerID="64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de" May 28 15:39:42.813151 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:42.813132 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de\": container with ID starting with 64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de not found: ID does not exist" containerID="64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de" May 28 15:39:42.813219 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.813164 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de"} err="failed to get container status \"64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de\": rpc error: code = NotFound desc = could not find container \"64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de\": container with ID starting with 64eb7002808d3f3151971e1a0bb2447c4a6bd8fd7a7f9940bdaaf1f6af5389de not found: ID does not exist" May 28 15:39:42.813219 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.813189 2569 scope.go:117] "RemoveContainer" containerID="a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573" May 28 15:39:42.813436 ip-10-0-130-98 kubenswrapper[2569]: E0528 15:39:42.813416 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573\": container with ID starting with a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573 not found: ID does not exist" containerID="a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573" May 28 15:39:42.813478 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:42.813443 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573"} err="failed to get container status \"a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573\": rpc error: code = NotFound desc = could not find container \"a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573\": container with ID starting with a287ea99fa7be5860d5e56b328006ade947fc0d9da59bba27e0e949943166573 not found: ID does not exist" May 28 15:39:43.422743 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:43.422717 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ls7p9_6ab0e46f-6489-46c5-b033-9a79dd389a4a/node-exporter/0.log" May 28 15:39:43.440196 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:43.440169 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ls7p9_6ab0e46f-6489-46c5-b033-9a79dd389a4a/kube-rbac-proxy/0.log" May 28 15:39:43.458245 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:43.458223 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ls7p9_6ab0e46f-6489-46c5-b033-9a79dd389a4a/init-textfile/0.log" May 28 15:39:44.073314 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:44.073282 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" path="/var/lib/kubelet/pods/4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72/volumes" May 28 15:39:46.574191 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.574160 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx"] May 28 15:39:46.574591 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.574377 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerName="gather" May 28 15:39:46.574591 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.574387 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerName="gather" May 28 15:39:46.574591 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.574399 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerName="copy" May 28 15:39:46.574591 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.574404 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerName="copy" May 28 15:39:46.574591 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.574455 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerName="copy" May 28 15:39:46.574591 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.574465 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d5c14ef-a4ec-4d0a-8509-3fc2faa5af72" containerName="gather" May 28 15:39:46.577848 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.577827 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.580064 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.580038 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lc5n8\"/\"openshift-service-ca.crt\"" May 28 15:39:46.580177 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.580040 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lc5n8\"/\"kube-root-ca.crt\"" May 28 15:39:46.580969 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.580954 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lc5n8\"/\"default-dockercfg-45tb4\"" May 28 15:39:46.583353 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.583328 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx"] May 28 15:39:46.680774 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.680730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-sys\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.680774 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.680779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-lib-modules\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.681007 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.680849 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-proc\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.681007 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.680911 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scw2v\" (UniqueName: \"kubernetes.io/projected/3b03bb0d-3328-4784-b3fd-405f4df54ee7-kube-api-access-scw2v\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.681007 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.680947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-podres\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782170 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782132 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-podres\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782181 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-sys\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782200 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-lib-modules\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782220 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-proc\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782246 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-scw2v\" (UniqueName: \"kubernetes.io/projected/3b03bb0d-3328-4784-b3fd-405f4df54ee7-kube-api-access-scw2v\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782273 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-sys\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782309 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-lib-modules\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782312 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-podres\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.782351 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.782338 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b03bb0d-3328-4784-b3fd-405f4df54ee7-proc\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.789273 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.789252 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scw2v\" (UniqueName: \"kubernetes.io/projected/3b03bb0d-3328-4784-b3fd-405f4df54ee7-kube-api-access-scw2v\") pod \"perf-node-gather-daemonset-wkcnx\" (UID: \"3b03bb0d-3328-4784-b3fd-405f4df54ee7\") " pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.842236 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.842156 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qp66_df4dc93c-2918-4976-a0a3-800fd9078a13/dns/0.log" May 28 15:39:46.861208 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.861180 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qp66_df4dc93c-2918-4976-a0a3-800fd9078a13/kube-rbac-proxy/0.log" May 28 15:39:46.888127 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.888104 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:46.991635 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:46.991591 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jlwmt_83f78757-e808-486a-9d6a-86aeccd930bb/dns-node-resolver/0.log" May 28 15:39:47.010649 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:47.010606 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx"] May 28 15:39:47.013448 ip-10-0-130-98 kubenswrapper[2569]: W0528 15:39:47.013418 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3b03bb0d_3328_4784_b3fd_405f4df54ee7.slice/crio-2f592c7f798ed1b84fcb8eb3b14c58b2489eb04e685556f8ae5244eea45db90d WatchSource:0}: Error finding container 2f592c7f798ed1b84fcb8eb3b14c58b2489eb04e685556f8ae5244eea45db90d: Status 404 returned error can't find the container with id 2f592c7f798ed1b84fcb8eb3b14c58b2489eb04e685556f8ae5244eea45db90d May 28 15:39:47.015063 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:47.015044 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 28 15:39:47.423279 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:47.423254 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w7rh6_d09e9953-ec38-42d4-ac0b-353cfdd3683d/node-ca/0.log" May 28 15:39:47.808770 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:47.808728 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" event={"ID":"3b03bb0d-3328-4784-b3fd-405f4df54ee7","Type":"ContainerStarted","Data":"41b5cda902c780f08898582876f2339b68d4e129ee790ecee75f6c2d415e987b"} May 28 15:39:47.808770 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:47.808766 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" event={"ID":"3b03bb0d-3328-4784-b3fd-405f4df54ee7","Type":"ContainerStarted","Data":"2f592c7f798ed1b84fcb8eb3b14c58b2489eb04e685556f8ae5244eea45db90d"} May 28 15:39:47.809223 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:47.808867 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:47.822774 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:47.822735 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" podStartSLOduration=1.8227228420000001 podStartE2EDuration="1.822722842s" podCreationTimestamp="2026-05-28 15:39:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-28 15:39:47.821543152 +0000 UTC m=+1760.324297312" watchObservedRunningTime="2026-05-28 15:39:47.822722842 +0000 UTC m=+1760.325477053" May 28 15:39:48.395605 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:48.395576 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggbmj_f7574eb4-3b1e-478d-9f61-dbf80303c7ea/serve-healthcheck-canary/0.log" May 28 15:39:48.704232 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:48.704135 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7wzv_7122333f-be47-41f9-96de-24884b2bca15/kube-rbac-proxy/0.log" May 28 15:39:48.722339 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:48.722307 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7wzv_7122333f-be47-41f9-96de-24884b2bca15/exporter/0.log" May 28 15:39:48.740452 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:48.740427 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7wzv_7122333f-be47-41f9-96de-24884b2bca15/extractor/0.log" May 28 15:39:53.533062 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.533032 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8vmcs_b848567f-fdd1-4b2c-82d2-f6692a9b4148/kube-multus-additional-cni-plugins/0.log" May 28 15:39:53.551111 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.551086 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8vmcs_b848567f-fdd1-4b2c-82d2-f6692a9b4148/egress-router-binary-copy/0.log" May 28 15:39:53.568610 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.568580 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8vmcs_b848567f-fdd1-4b2c-82d2-f6692a9b4148/cni-plugins/0.log" May 28 15:39:53.586119 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.586100 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8vmcs_b848567f-fdd1-4b2c-82d2-f6692a9b4148/bond-cni-plugin/0.log" May 28 15:39:53.603248 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.603230 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8vmcs_b848567f-fdd1-4b2c-82d2-f6692a9b4148/routeoverride-cni/0.log" May 28 15:39:53.622594 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.622570 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8vmcs_b848567f-fdd1-4b2c-82d2-f6692a9b4148/whereabouts-cni-bincopy/0.log" May 28 15:39:53.640136 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.640115 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8vmcs_b848567f-fdd1-4b2c-82d2-f6692a9b4148/whereabouts-cni/0.log" May 28 15:39:53.823110 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:53.823036 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lc5n8/perf-node-gather-daemonset-wkcnx" May 28 15:39:54.001660 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.001634 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mndtq_33dd655e-c457-41ba-b059-59c44998d358/kube-multus/0.log" May 28 15:39:54.060279 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.060256 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ckhtc_70961166-700d-40ad-867d-dcdf297e6704/network-metrics-daemon/0.log" May 28 15:39:54.078690 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.078607 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ckhtc_70961166-700d-40ad-867d-dcdf297e6704/kube-rbac-proxy/0.log" May 28 15:39:54.789712 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.789684 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-controller/0.log" May 28 15:39:54.806589 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.806565 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/0.log" May 28 15:39:54.813934 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.813918 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovn-acl-logging/1.log" May 28 15:39:54.831288 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.831266 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/kube-rbac-proxy-node/0.log" May 28 15:39:54.851337 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.851311 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/kube-rbac-proxy-ovn-metrics/0.log" May 28 15:39:54.869250 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.869229 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/northd/0.log" May 28 15:39:54.890924 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.890425 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/nbdb/0.log" May 28 15:39:54.910132 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.910103 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/sbdb/0.log" May 28 15:39:54.999336 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:54.999307 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gx9nw_67fbf203-2dbe-401e-9d41-b9e841caa1e8/ovnkube-controller/0.log" May 28 15:39:56.535826 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:56.535800 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2vt4s_e018896b-bc34-4bda-b761-752548a2cde1/network-check-target-container/0.log" May 28 15:39:57.455764 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:57.455730 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-l6d25_e5cb0634-a302-4dfd-a99d-5de8107704dc/iptables-alerter/0.log" May 28 15:39:58.043264 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:39:58.043236 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zsdb5_5352d147-37e5-4291-9fe9-66faf7e057c2/tuned/0.log" May 28 15:40:01.037552 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:40:01.037528 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-sq46t_ed77076b-c4ca-4f11-934e-1f1ed7350613/csi-driver/0.log" May 28 15:40:01.055260 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:40:01.055238 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-sq46t_ed77076b-c4ca-4f11-934e-1f1ed7350613/csi-node-driver-registrar/0.log" May 28 15:40:01.073466 ip-10-0-130-98 kubenswrapper[2569]: I0528 15:40:01.073441 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-sq46t_ed77076b-c4ca-4f11-934e-1f1ed7350613/csi-liveness-probe/0.log"