Jun 10 10:44:58.240380 ip-10-0-128-191 systemd[1]: Starting Kubernetes Kubelet... Jun 10 10:44:58.616381 ip-10-0-128-191 kubenswrapper[2545]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 10 10:44:58.616381 ip-10-0-128-191 kubenswrapper[2545]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jun 10 10:44:58.616381 ip-10-0-128-191 kubenswrapper[2545]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 10 10:44:58.616955 ip-10-0-128-191 kubenswrapper[2545]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jun 10 10:44:58.616955 ip-10-0-128-191 kubenswrapper[2545]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 10 10:44:58.619389 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.619314 2545 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jun 10 10:44:58.624097 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624082 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:58.624097 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624096 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624099 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624102 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624107 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624112 2545 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624115 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624118 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624121 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624124 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624127 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624130 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624132 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624138 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624141 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624144 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624146 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624149 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624151 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624154 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:58.624161 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624156 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624159 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624161 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624164 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624167 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624170 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624173 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624175 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624178 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624180 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624184 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624186 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624189 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624191 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624194 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624196 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624199 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624202 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624205 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624207 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:58.624601 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624210 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624212 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624214 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624217 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624219 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624222 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624224 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624227 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624229 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624231 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624234 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624236 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624239 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624241 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624244 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624250 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624253 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624256 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624259 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:58.625081 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624262 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624265 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624268 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624270 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624273 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624277 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624280 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624282 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624284 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624287 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624289 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624292 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624294 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624297 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624299 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624301 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624304 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624306 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624309 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:58.625535 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624311 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624313 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624316 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624318 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624321 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624323 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624325 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624328 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624331 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624334 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624336 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624339 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624341 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624344 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624346 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624349 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624351 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624354 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624356 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624358 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:58.625992 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.624361 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:58.626779 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626766 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:58.626779 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626778 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626783 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626786 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626796 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626800 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626803 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626806 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626809 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626812 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626814 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626817 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626820 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626822 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626825 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626827 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626830 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626832 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626835 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626838 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:58.626831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626841 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626843 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626846 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626849 2545 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626851 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626854 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626856 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626859 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626861 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626864 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626866 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626869 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626872 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626874 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626877 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626879 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626881 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626889 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626892 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626894 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:58.627330 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626897 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626899 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626901 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626904 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626906 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626908 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626912 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626915 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626918 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626920 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626923 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626925 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626927 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626930 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626934 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626936 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626939 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626941 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626944 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:58.627811 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626946 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626949 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626952 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626954 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626956 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626972 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626975 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626977 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626980 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626982 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626985 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626994 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626996 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.626999 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627006 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627008 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627011 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627014 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627016 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627018 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:58.628264 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627021 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627024 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627026 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627028 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627031 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627033 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627037 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627041 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627044 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627047 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627050 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627053 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627055 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627058 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627060 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627063 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627065 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627068 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627086 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:58.628738 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627090 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627510 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627516 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627519 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627528 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627531 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627534 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627537 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627539 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627542 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627544 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627547 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627550 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627553 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627556 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627559 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627561 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627564 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627566 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:58.629201 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627568 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627572 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627575 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627577 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627580 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627584 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627587 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627590 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627593 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627596 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627598 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627601 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627603 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627606 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627608 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627610 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627613 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627615 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627623 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627626 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:58.629643 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627628 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627630 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627633 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627636 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627639 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627642 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627644 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627647 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627649 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627652 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627654 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627657 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627659 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627662 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627664 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627667 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627669 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627672 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627674 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627677 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:58.630129 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627679 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627681 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627684 2545 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627686 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627689 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627691 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627695 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627697 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627700 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627702 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627705 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627713 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627715 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627717 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627720 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627722 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627725 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627727 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627730 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627732 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:58.630617 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627735 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627737 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627739 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627742 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627745 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627747 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627750 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627752 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627755 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627757 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627759 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627762 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627764 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627767 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627769 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627773 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627776 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627779 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627781 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:58.631088 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627784 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.627787 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627848 2545 flags.go:64] FLAG: --address="0.0.0.0" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627859 2545 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627870 2545 flags.go:64] FLAG: --anonymous-auth="true" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627876 2545 flags.go:64] FLAG: --application-metrics-count-limit="100" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627880 2545 flags.go:64] FLAG: --authentication-token-webhook="false" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627883 2545 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627886 2545 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627890 2545 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627893 2545 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627896 2545 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627899 2545 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627903 2545 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627906 2545 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627908 2545 flags.go:64] FLAG: --cgroup-root="" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627911 2545 flags.go:64] FLAG: --cgroups-per-qos="true" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627914 2545 flags.go:64] FLAG: --client-ca-file="" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627917 2545 flags.go:64] FLAG: --cloud-provider="external" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627920 2545 flags.go:64] FLAG: --cluster-dns="[]" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627927 2545 flags.go:64] FLAG: --cluster-domain="" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627930 2545 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627933 2545 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627936 2545 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jun 10 10:44:58.631514 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627939 2545 flags.go:64] FLAG: --container-log-max-files="5" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627943 2545 flags.go:64] FLAG: --container-log-max-size="10Mi" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627945 2545 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627948 2545 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627952 2545 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627954 2545 flags.go:64] FLAG: --contention-profiling="false" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627957 2545 flags.go:64] FLAG: --cpu-cfs-quota="true" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627974 2545 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627977 2545 flags.go:64] FLAG: --cpu-manager-policy="none" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627980 2545 flags.go:64] FLAG: --cpu-manager-policy-options="" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627986 2545 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627989 2545 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627991 2545 flags.go:64] FLAG: --enable-debugging-handlers="true" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.627994 2545 flags.go:64] FLAG: --enable-load-reader="false" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628005 2545 flags.go:64] FLAG: --enable-server="true" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628008 2545 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628014 2545 flags.go:64] FLAG: --event-burst="100" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628017 2545 flags.go:64] FLAG: --event-qps="50" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628020 2545 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628023 2545 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628026 2545 flags.go:64] FLAG: --eviction-hard="" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628029 2545 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628032 2545 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628035 2545 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628038 2545 flags.go:64] FLAG: --eviction-soft="" Jun 10 10:44:58.632121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628041 2545 flags.go:64] FLAG: --eviction-soft-grace-period="" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628043 2545 flags.go:64] FLAG: --exit-on-lock-contention="false" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628046 2545 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628049 2545 flags.go:64] FLAG: --experimental-mounter-path="" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628052 2545 flags.go:64] FLAG: --fail-cgroupv1="false" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628054 2545 flags.go:64] FLAG: --fail-swap-on="true" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628057 2545 flags.go:64] FLAG: --feature-gates="" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628060 2545 flags.go:64] FLAG: --file-check-frequency="20s" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628063 2545 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628066 2545 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628069 2545 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628072 2545 flags.go:64] FLAG: --healthz-port="10248" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628074 2545 flags.go:64] FLAG: --help="false" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628077 2545 flags.go:64] FLAG: --hostname-override="ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628080 2545 flags.go:64] FLAG: --housekeeping-interval="10s" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628082 2545 flags.go:64] FLAG: --http-check-frequency="20s" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628085 2545 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628089 2545 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628092 2545 flags.go:64] FLAG: --image-gc-high-threshold="85" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628095 2545 flags.go:64] FLAG: --image-gc-low-threshold="80" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628098 2545 flags.go:64] FLAG: --image-service-endpoint="" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628100 2545 flags.go:64] FLAG: --kernel-memcg-notification="false" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628111 2545 flags.go:64] FLAG: --kube-api-burst="100" Jun 10 10:44:58.632691 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628114 2545 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628117 2545 flags.go:64] FLAG: --kube-api-qps="50" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628119 2545 flags.go:64] FLAG: --kube-reserved="" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628122 2545 flags.go:64] FLAG: --kube-reserved-cgroup="" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628124 2545 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628127 2545 flags.go:64] FLAG: --kubelet-cgroups="" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628130 2545 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628133 2545 flags.go:64] FLAG: --lock-file="" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628135 2545 flags.go:64] FLAG: --log-cadvisor-usage="false" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628138 2545 flags.go:64] FLAG: --log-flush-frequency="5s" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628141 2545 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628146 2545 flags.go:64] FLAG: --log-json-split-stream="false" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628149 2545 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628151 2545 flags.go:64] FLAG: --log-text-split-stream="false" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628154 2545 flags.go:64] FLAG: --logging-format="text" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628157 2545 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628160 2545 flags.go:64] FLAG: --make-iptables-util-chains="true" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628162 2545 flags.go:64] FLAG: --manifest-url="" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628165 2545 flags.go:64] FLAG: --manifest-url-header="" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628171 2545 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628174 2545 flags.go:64] FLAG: --max-open-files="1000000" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628178 2545 flags.go:64] FLAG: --max-pods="110" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628181 2545 flags.go:64] FLAG: --maximum-dead-containers="-1" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628183 2545 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628186 2545 flags.go:64] FLAG: --memory-manager-policy="None" Jun 10 10:44:58.633321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628189 2545 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628194 2545 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628196 2545 flags.go:64] FLAG: --node-ip="0.0.0.0" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628199 2545 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628208 2545 flags.go:64] FLAG: --node-status-max-images="50" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628211 2545 flags.go:64] FLAG: --node-status-update-frequency="10s" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628214 2545 flags.go:64] FLAG: --oom-score-adj="-999" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628222 2545 flags.go:64] FLAG: --pod-cidr="" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628228 2545 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746976bfb3a92b93dacc79c151b98fd4c932eef19c3e2c1516b14eebcac9184d" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628232 2545 flags.go:64] FLAG: --pod-manifest-path="" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628235 2545 flags.go:64] FLAG: --pod-max-pids="-1" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628238 2545 flags.go:64] FLAG: --pods-per-core="0" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628241 2545 flags.go:64] FLAG: --port="10250" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628243 2545 flags.go:64] FLAG: --protect-kernel-defaults="false" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628246 2545 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07613feece616ee3b" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628249 2545 flags.go:64] FLAG: --qos-reserved="" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628252 2545 flags.go:64] FLAG: --read-only-port="10255" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628255 2545 flags.go:64] FLAG: --register-node="true" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628257 2545 flags.go:64] FLAG: --register-with-taints="" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628261 2545 flags.go:64] FLAG: --registry-burst="10" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628263 2545 flags.go:64] FLAG: --registry-qps="5" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628266 2545 flags.go:64] FLAG: --reserved-cpus="" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628269 2545 flags.go:64] FLAG: --reserved-memory="" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628273 2545 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628276 2545 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jun 10 10:44:58.633929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628284 2545 flags.go:64] FLAG: --rotate-certificates="false" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628287 2545 flags.go:64] FLAG: --rotate-server-certificates="false" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628290 2545 flags.go:64] FLAG: --runonce="false" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628293 2545 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628296 2545 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628299 2545 flags.go:64] FLAG: --seccomp-default="false" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628301 2545 flags.go:64] FLAG: --serialize-image-pulls="true" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628304 2545 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628308 2545 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628311 2545 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628314 2545 flags.go:64] FLAG: --storage-driver-password="root" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628317 2545 flags.go:64] FLAG: --storage-driver-secure="false" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628319 2545 flags.go:64] FLAG: --storage-driver-table="stats" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628322 2545 flags.go:64] FLAG: --storage-driver-user="root" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628324 2545 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628333 2545 flags.go:64] FLAG: --sync-frequency="1m0s" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628337 2545 flags.go:64] FLAG: --system-cgroups="" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628339 2545 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628344 2545 flags.go:64] FLAG: --system-reserved-cgroup="" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628347 2545 flags.go:64] FLAG: --tls-cert-file="" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628349 2545 flags.go:64] FLAG: --tls-cipher-suites="[]" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628356 2545 flags.go:64] FLAG: --tls-min-version="" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628358 2545 flags.go:64] FLAG: --tls-private-key-file="" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628361 2545 flags.go:64] FLAG: --topology-manager-policy="none" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628364 2545 flags.go:64] FLAG: --topology-manager-policy-options="" Jun 10 10:44:58.634543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628366 2545 flags.go:64] FLAG: --topology-manager-scope="container" Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628369 2545 flags.go:64] FLAG: --v="2" Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628373 2545 flags.go:64] FLAG: --version="false" Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628377 2545 flags.go:64] FLAG: --vmodule="" Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628380 2545 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.628383 2545 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628551 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628555 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628558 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628561 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628564 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628567 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628569 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628572 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628574 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628578 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628581 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628584 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628586 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628588 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628591 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:58.635195 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628593 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628596 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628606 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628609 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628611 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628613 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628616 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628618 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628621 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628623 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628626 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628629 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628631 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628633 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628636 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628638 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628641 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628643 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628646 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628648 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:58.635695 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628651 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628654 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628658 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628661 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628663 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628666 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628670 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628672 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628675 2545 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628678 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628680 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628682 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628684 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628687 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628691 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628699 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628701 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628704 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628706 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:58.636293 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628709 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628711 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628714 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628716 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628719 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628721 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628724 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628727 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628730 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628733 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628736 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628739 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628742 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628745 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628747 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628750 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628753 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628755 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628758 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628762 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:58.637069 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628764 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628767 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628769 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628772 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628776 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628778 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628781 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628785 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628788 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628796 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628799 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628801 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628804 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628806 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628809 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628811 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628814 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628816 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628819 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:58.637556 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628821 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:58.638024 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628824 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:58.638024 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628826 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:58.638024 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628829 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:58.638024 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628831 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:58.638024 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.628833 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:58.638024 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.629706 2545 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 10 10:44:58.638176 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.638156 2545 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jun 10 10:44:58.638176 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.638170 2545 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jun 10 10:44:58.639043 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.639032 2545 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jun 10 10:44:58.639075 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.639043 2545 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jun 10 10:44:58.639120 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639112 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:58.639120 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639119 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639123 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639127 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639130 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639132 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639135 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639138 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639140 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639143 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639146 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639149 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639151 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639154 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639157 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639159 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639162 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639164 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639167 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639169 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639172 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:58.639177 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639174 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639177 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639179 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639182 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639185 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639188 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639191 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639194 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639196 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639199 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639201 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639204 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639206 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639209 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639211 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639214 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639216 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639219 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639221 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639224 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:58.639644 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639226 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639230 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639233 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639237 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639239 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639242 2545 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639244 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639247 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639249 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639252 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639254 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639257 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639259 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639262 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639265 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639268 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639271 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639273 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639276 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639279 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:58.640157 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639281 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639284 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639286 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639289 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639291 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639294 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639297 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639299 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639302 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639304 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639307 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639309 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639312 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639314 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639317 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639319 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639322 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639324 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639334 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:58.640630 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639338 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639340 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639343 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639346 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639350 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639353 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639356 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639359 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639361 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639364 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639366 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639369 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639372 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639374 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639377 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639379 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639382 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639384 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:58.641084 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639386 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.639391 2545 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639505 2545 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639510 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639513 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639516 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639518 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639521 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639524 2545 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639526 2545 feature_gate.go:328] unrecognized feature gate: Example2 Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639529 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639531 2545 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639534 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639536 2545 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 10 10:44:58.641505 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639545 2545 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639549 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639552 2545 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639555 2545 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639558 2545 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639561 2545 feature_gate.go:328] unrecognized feature gate: Example Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639564 2545 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639567 2545 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639571 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639573 2545 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639576 2545 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639579 2545 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639581 2545 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639584 2545 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639587 2545 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639589 2545 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639592 2545 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639595 2545 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639598 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639600 2545 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 10 10:44:58.641831 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639602 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639604 2545 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639607 2545 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639609 2545 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639611 2545 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639614 2545 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639616 2545 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639618 2545 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639621 2545 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639623 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639625 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639628 2545 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639630 2545 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639642 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639645 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639647 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639650 2545 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639653 2545 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639655 2545 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639659 2545 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 10 10:44:58.642314 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639661 2545 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639664 2545 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639666 2545 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639669 2545 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639671 2545 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639674 2545 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639676 2545 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639679 2545 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639681 2545 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639684 2545 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639686 2545 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639689 2545 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639691 2545 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639693 2545 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639696 2545 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639698 2545 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639701 2545 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639703 2545 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639706 2545 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639708 2545 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 10 10:44:58.642774 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639710 2545 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639713 2545 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639715 2545 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639717 2545 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639720 2545 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639722 2545 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639725 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639729 2545 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639732 2545 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639735 2545 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639737 2545 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639740 2545 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639742 2545 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639744 2545 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639747 2545 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639749 2545 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639751 2545 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639754 2545 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639756 2545 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639758 2545 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 10 10:44:58.643255 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639761 2545 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639763 2545 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639765 2545 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639768 2545 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639770 2545 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639774 2545 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:58.639777 2545 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.639782 2545 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.639877 2545 server.go:962] "Client rotation is on, will bootstrap in background" Jun 10 10:44:58.643702 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.643640 2545 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jun 10 10:44:58.645266 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.645252 2545 server.go:1019] "Starting client certificate rotation" Jun 10 10:44:58.645370 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.645353 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 10 10:44:58.645398 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.645392 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 10 10:44:58.648770 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.648756 2545 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 10 10:44:58.652755 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.652737 2545 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 10 10:44:58.663942 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.663925 2545 log.go:25] "Validated CRI v1 runtime API" Jun 10 10:44:58.666818 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.666805 2545 log.go:25] "Validated CRI v1 image API" Jun 10 10:44:58.667600 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.667584 2545 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jun 10 10:44:58.670117 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.670095 2545 fs.go:135] Filesystem UUIDs: map[319dbc1e-bd59-44a6-8d8c-b2989edbfa18:/dev/nvme0n1p3 36522b06-625a-46b7-9fef-93a4c7b9c6eb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jun 10 10:44:58.670177 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.670115 2545 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jun 10 10:44:58.675545 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.675525 2545 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 10 10:44:58.676579 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.676464 2545 manager.go:216] Machine: {Timestamp:2026-06-10 10:44:58.674582978 +0000 UTC m=+0.323515513 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199416 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a61b52edad5fcb1485145e27aa51d SystemUUID:ec2a61b5-2eda-d5fc-b148-5145e27aa51d BootID:f7f822a6-8e6e-4cd5-a8f0-fa95bc9baa16 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:7b:05:41:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:7b:05:41:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:3a:c7:b1:07:c9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jun 10 10:44:58.676579 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.676565 2545 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jun 10 10:44:58.676723 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.676646 2545 manager.go:232] Version: {KernelVersion:5.14.0-570.118.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260602-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jun 10 10:44:58.680205 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.680171 2545 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jun 10 10:44:58.680361 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.680209 2545 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jun 10 10:44:58.681045 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.681033 2545 topology_manager.go:138] "Creating topology manager with none policy" Jun 10 10:44:58.681105 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.681050 2545 container_manager_linux.go:309] "Creating device plugin manager" Jun 10 10:44:58.681105 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.681059 2545 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 10 10:44:58.681105 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.681079 2545 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 10 10:44:58.682244 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.682232 2545 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jun 10 10:44:58.686537 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.686524 2545 state_mem.go:36] "Initialized new in-memory state store" Jun 10 10:44:58.686665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.686654 2545 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jun 10 10:44:58.686732 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.686688 2545 kubelet.go:486] "Attempting to sync node with API server" Jun 10 10:44:58.686732 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.686702 2545 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jun 10 10:44:58.686732 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.686720 2545 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jun 10 10:44:58.686732 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.686732 2545 kubelet.go:398] "Adding apiserver pod source" Jun 10 10:44:58.686887 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.686744 2545 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jun 10 10:44:58.687093 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.687080 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 10 10:44:58.687191 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.687102 2545 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 10 10:44:58.689651 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.689636 2545 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jun 10 10:44:58.691282 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.691270 2545 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jun 10 10:44:58.691321 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.691298 2545 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jun 10 10:44:58.692715 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692703 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jun 10 10:44:58.692715 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692718 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692724 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692730 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692739 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692744 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692752 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692757 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692764 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692769 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692777 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jun 10 10:44:58.692809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.692788 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jun 10 10:44:58.693683 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.693673 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jun 10 10:44:58.693683 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.693683 2545 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jun 10 10:44:58.696016 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.695995 2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qv7q5" Jun 10 10:44:58.696796 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.696783 2545 server.go:1268] "Started kubelet" Jun 10 10:44:58.696929 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.696892 2545 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jun 10 10:44:58.697255 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.697242 2545 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jun 10 10:44:58.697316 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.697269 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jun 10 10:44:58.697363 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.697342 2545 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jun 10 10:44:58.697818 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.697768 2545 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jun 10 10:44:58.697877 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.697843 2545 server_v1.go:49] "podresources" method="list" useActivePods=true Jun 10 10:44:58.699203 ip-10-0-128-191 systemd[1]: Started Kubernetes Kubelet. Jun 10 10:44:58.700069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.699996 2545 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jun 10 10:44:58.700530 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.700514 2545 server.go:310] "Adding debug handlers to kubelet server" Jun 10 10:44:58.703772 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.701851 2545 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-191.ec2.internal.18b7b25efd6cf058 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-191.ec2.internal,UID:ip-10-0-128-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-191.ec2.internal,},FirstTimestamp:2026-06-10 10:44:58.696765528 +0000 UTC m=+0.345698060,LastTimestamp:2026-06-10 10:44:58.696765528 +0000 UTC m=+0.345698060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-191.ec2.internal,}" Jun 10 10:44:58.704251 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.704238 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qv7q5" Jun 10 10:44:58.704849 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.704831 2545 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jun 10 10:44:58.707389 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.707377 2545 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jun 10 10:44:58.707429 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.707388 2545 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jun 10 10:44:58.707566 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.707552 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:58.708197 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.708180 2545 volume_manager.go:311] "The desired_state_of_world populator starts" Jun 10 10:44:58.708315 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.708258 2545 volume_manager.go:313] "Starting Kubelet Volume Manager" Jun 10 10:44:58.708315 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.708201 2545 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jun 10 10:44:58.708421 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.708388 2545 reconstruct.go:97] "Volume reconstruction finished" Jun 10 10:44:58.708421 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.708396 2545 reconciler.go:29] "Reconciler: start to sync state" Jun 10 10:44:58.710979 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.710941 2545 factory.go:55] Registering systemd factory Jun 10 10:44:58.710979 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.710979 2545 factory.go:223] Registration of the systemd container factory successfully Jun 10 10:44:58.711335 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.711320 2545 factory.go:153] Registering CRI-O factory Jun 10 10:44:58.711391 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.711339 2545 factory.go:223] Registration of the crio container factory successfully Jun 10 10:44:58.711391 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.711378 2545 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jun 10 10:44:58.711483 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.711399 2545 factory.go:103] Registering Raw factory Jun 10 10:44:58.711483 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.711427 2545 manager.go:1196] Started watching for new ooms in manager Jun 10 10:44:58.711997 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.711982 2545 manager.go:349] Starting recovery of all containers Jun 10 10:44:58.719776 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.719624 2545 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:58.723207 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.723187 2545 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-191.ec2.internal\" not found" node="ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.723616 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.723602 2545 manager.go:354] Recovery completed Jun 10 10:44:58.727663 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.727651 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:58.729832 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.729815 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:58.729909 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.729849 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:58.729909 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.729860 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:58.730368 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.730353 2545 cpu_manager.go:222] "Starting CPU manager" policy="none" Jun 10 10:44:58.730368 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.730364 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:58.730368 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.730367 2545 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jun 10 10:44:58.730504 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.730386 2545 state_mem.go:36] "Initialized new in-memory state store" Jun 10 10:44:58.731023 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.731006 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:58.731100 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.731031 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:58.731100 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.731041 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:58.732343 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.732331 2545 policy_none.go:49] "None policy: Start" Jun 10 10:44:58.732382 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.732346 2545 memory_manager.go:187] "Starting memorymanager" policy="None" Jun 10 10:44:58.732382 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.732356 2545 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jun 10 10:44:58.733865 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.733854 2545 policy_none.go:47] "Start" Jun 10 10:44:58.766534 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.766520 2545 manager.go:337] "Starting Device Plugin manager" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.766553 2545 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.766563 2545 server.go:87] "Starting device plugin registration server" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.766795 2545 eviction_manager.go:189] "Eviction manager: starting control loop" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.766806 2545 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.766899 2545 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.767049 2545 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.767062 2545 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.767565 2545 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.767605 2545 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.774902 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.776053 2545 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.776085 2545 status_manager.go:244] "Starting to sync pod status with apiserver" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.776103 2545 kubelet.go:2448] "Starting kubelet main sync loop" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.776139 2545 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jun 10 10:44:58.785773 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.778245 2545 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:58.867622 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.867569 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:58.868393 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.868379 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:58.868478 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.868406 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:58.868478 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.868415 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:58.868478 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.868435 2545 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.876856 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.876841 2545 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.876942 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.876864 2545 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-191.ec2.internal\": node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:58.876942 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.876938 2545 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"] Jun 10 10:44:58.877060 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.877006 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:58.877678 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.877664 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:58.877761 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.877692 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:58.877761 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.877705 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:58.880079 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880065 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:58.880175 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880161 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.880219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880188 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:58.880900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880885 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:58.880989 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880901 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:58.880989 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880911 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:58.880989 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880924 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:58.880989 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880924 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:58.881115 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.880992 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:58.883181 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.883165 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.883267 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.883190 2545 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 10 10:44:58.883795 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.883779 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory" Jun 10 10:44:58.883854 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.883805 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure" Jun 10 10:44:58.883854 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.883814 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID" Jun 10 10:44:58.890725 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.890711 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:58.898468 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.898451 2545 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-191.ec2.internal\" not found" node="ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.901548 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.901533 2545 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-191.ec2.internal\" not found" node="ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.909710 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.909694 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e8a60c6729dc0922027f4cd7063603-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"29e8a60c6729dc0922027f4cd7063603\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.909791 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.909718 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75bf6497d56e225e5bd01fc8d3fdd351-config\") pod \"kube-apiserver-proxy-ip-10-0-128-191.ec2.internal\" (UID: \"75bf6497d56e225e5bd01fc8d3fdd351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.909791 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:58.909734 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29e8a60c6729dc0922027f4cd7063603-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"29e8a60c6729dc0922027f4cd7063603\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:58.990888 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:58.990866 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.010579 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.010560 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e8a60c6729dc0922027f4cd7063603-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"29e8a60c6729dc0922027f4cd7063603\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.010643 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.010583 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75bf6497d56e225e5bd01fc8d3fdd351-config\") pod \"kube-apiserver-proxy-ip-10-0-128-191.ec2.internal\" (UID: \"75bf6497d56e225e5bd01fc8d3fdd351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.010643 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.010606 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29e8a60c6729dc0922027f4cd7063603-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"29e8a60c6729dc0922027f4cd7063603\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.010742 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.010644 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29e8a60c6729dc0922027f4cd7063603-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"29e8a60c6729dc0922027f4cd7063603\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.010742 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.010667 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29e8a60c6729dc0922027f4cd7063603-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"29e8a60c6729dc0922027f4cd7063603\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.010742 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.010693 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75bf6497d56e225e5bd01fc8d3fdd351-config\") pod \"kube-apiserver-proxy-ip-10-0-128-191.ec2.internal\" (UID: \"75bf6497d56e225e5bd01fc8d3fdd351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.091169 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.091141 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.191819 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.191766 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.205904 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.205888 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.210105 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.209956 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.292769 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.292741 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.393197 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.393174 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.493614 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.493558 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.594168 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.594145 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.644653 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.644633 2545 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jun 10 10:44:59.645284 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.644768 2545 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 10 10:44:59.645284 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.644813 2545 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 10 10:44:59.695109 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.695083 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.711244 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.711217 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-09 10:39:58 +0000 UTC" deadline="2028-03-20 09:18:43.224595888 +0000 UTC" Jun 10 10:44:59.711324 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.711241 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15574h33m43.513357403s" Jun 10 10:44:59.728515 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.728497 2545 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:59.751468 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:59.751416 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e8a60c6729dc0922027f4cd7063603.slice/crio-08e79d7d69f7adb3b2955a9c3d30da1f805401d91733b750776d44eb23fd420c WatchSource:0}: Error finding container 08e79d7d69f7adb3b2955a9c3d30da1f805401d91733b750776d44eb23fd420c: Status 404 returned error can't find the container with id 08e79d7d69f7adb3b2955a9c3d30da1f805401d91733b750776d44eb23fd420c Jun 10 10:44:59.751868 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:44:59.751849 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bf6497d56e225e5bd01fc8d3fdd351.slice/crio-99bb3557e8d19924021fa844014472d1b571ed0c9435d4d822269250f0c912cc WatchSource:0}: Error finding container 99bb3557e8d19924021fa844014472d1b571ed0c9435d4d822269250f0c912cc: Status 404 returned error can't find the container with id 99bb3557e8d19924021fa844014472d1b571ed0c9435d4d822269250f0c912cc Jun 10 10:44:59.758248 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.758229 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 10 10:44:59.779491 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.779456 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" event={"ID":"29e8a60c6729dc0922027f4cd7063603","Type":"ContainerStarted","Data":"08e79d7d69f7adb3b2955a9c3d30da1f805401d91733b750776d44eb23fd420c"} Jun 10 10:44:59.780466 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.780448 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" event={"ID":"75bf6497d56e225e5bd01fc8d3fdd351","Type":"ContainerStarted","Data":"99bb3557e8d19924021fa844014472d1b571ed0c9435d4d822269250f0c912cc"} Jun 10 10:44:59.795591 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:44:59.795570 2545 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found" Jun 10 10:44:59.821941 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.821924 2545 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:44:59.908071 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.908054 2545 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.922257 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.922238 2545 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 10 10:44:59.923390 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.923379 2545 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" Jun 10 10:44:59.932296 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:44:59.932284 2545 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 10 10:45:00.687556 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.687512 2545 apiserver.go:52] "Watching apiserver" Jun 10 10:45:00.696607 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.696585 2545 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jun 10 10:45:00.697032 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.697006 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-47m9g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-cxkhd","openshift-network-diagnostics/network-check-target-gdqfd","openshift-network-operator/iptables-alerter-498lq","openshift-ovn-kubernetes/ovnkube-node-pfkh8","kube-system/konnectivity-agent-ql89z","kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc","openshift-cluster-node-tuning-operator/tuned-p82p8","openshift-dns/node-resolver-p4l5s","openshift-image-registry/node-ca-jgp8h","openshift-multus/multus-h4kch","openshift-multus/network-metrics-daemon-6xj9p"] Jun 10 10:45:00.699808 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.699780 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:00.701866 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.701838 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:00.701979 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.701924 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:00.703012 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.702992 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jun 10 10:45:00.703111 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.703027 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jun 10 10:45:00.703111 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.703074 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r7np4\"" Jun 10 10:45:00.704046 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.704016 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.706360 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.706336 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jun 10 10:45:00.706465 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.706343 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.706732 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.706716 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s52c9\"" Jun 10 10:45:00.706813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.706736 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jun 10 10:45:00.706813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.706783 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jun 10 10:45:00.706906 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.706850 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jun 10 10:45:00.706954 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.706929 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jun 10 10:45:00.708175 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.708048 2545 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jun 10 10:45:00.708676 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.708654 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.708765 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.708717 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5hrp7\"" Jun 10 10:45:00.709182 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.709165 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jun 10 10:45:00.710772 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.710753 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.711083 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.711060 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jun 10 10:45:00.711199 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.711177 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jun 10 10:45:00.711423 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.711404 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r2dzb\"" Jun 10 10:45:00.711423 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.711417 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jun 10 10:45:00.713823 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.713791 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:00.713911 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.713857 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:00.714012 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.713993 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jun 10 10:45:00.714222 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.714175 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lhc82\"" Jun 10 10:45:00.714222 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.714190 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jun 10 10:45:00.716551 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.716534 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.718589 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.718492 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.719434 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.719578 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.719602 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.719810 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhp5\" (UniqueName: \"kubernetes.io/projected/452be7ad-2d9a-4448-899a-d5e68cc10f2e-kube-api-access-tmhp5\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.719843 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b01aab23-af40-42ef-914f-2ca0ea129769-multus-daemon-config\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.719869 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-netns\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720057 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-cni-bin\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720089 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6845d82a-8317-441b-b59a-41bc1b86c9df-host-slash\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720113 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w7lf2\"" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720118 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/637ed0ce-8721-46e2-8c84-bd306db605c0-konnectivity-ca\") pod \"konnectivity-agent-ql89z\" (UID: \"637ed0ce-8721-46e2-8c84-bd306db605c0\") " pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720168 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-cnibin\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720115 2545 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720269 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qplf4\" (UniqueName: \"kubernetes.io/projected/db8e86ee-d321-4fdc-be9a-495560a2f399-kube-api-access-qplf4\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720313 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-os-release\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720342 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-k8s-cni-cncf-io\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720379 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtp7\" (UniqueName: \"kubernetes.io/projected/b01aab23-af40-42ef-914f-2ca0ea129769-kube-api-access-swtp7\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720417 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720549 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-system-cni-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.721279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720622 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-cni-multus\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720684 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-conf-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720719 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgrk\" (UniqueName: \"kubernetes.io/projected/6845d82a-8317-441b-b59a-41bc1b86c9df-kube-api-access-ftgrk\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720749 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720786 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-cni-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720807 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-cnibin\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720823 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-etc-kubernetes\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720839 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/637ed0ce-8721-46e2-8c84-bd306db605c0-agent-certs\") pod \"konnectivity-agent-ql89z\" (UID: \"637ed0ce-8721-46e2-8c84-bd306db605c0\") " pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720858 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720877 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720896 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-socket-dir-parent\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720915 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-kubelet\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720935 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720933 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6845d82a-8317-441b-b59a-41bc1b86c9df-iptables-alerter-script\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.720982 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-os-release\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721001 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b01aab23-af40-42ef-914f-2ca0ea129769-cni-binary-copy\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721031 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:00.722219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721063 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-system-cni-dir\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.722939 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721349 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-hostroot\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722939 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721389 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-multus-certs\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.722939 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721572 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-khtsb\"" Jun 10 10:45:00.722939 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721597 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jun 10 10:45:00.722939 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.721917 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.723930 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.723911 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.725836 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.725815 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jun 10 10:45:00.725836 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.725842 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jun 10 10:45:00.726025 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.726015 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jun 10 10:45:00.726096 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.726049 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hg2vt\"" Jun 10 10:45:00.726096 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.726061 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jun 10 10:45:00.726274 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.726200 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jun 10 10:45:00.726419 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.726404 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.726497 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.726462 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:00.726626 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.726585 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jun 10 10:45:00.727124 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.727103 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jun 10 10:45:00.727324 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.727306 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c44m8\"" Jun 10 10:45:00.727755 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.727603 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jun 10 10:45:00.727755 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.727710 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jun 10 10:45:00.740571 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.740549 2545 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lgvmp" Jun 10 10:45:00.748642 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.748618 2545 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lgvmp" Jun 10 10:45:00.806063 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.806039 2545 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:45:00.809687 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.809668 2545 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jun 10 10:45:00.822116 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822088 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-kubernetes\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.822221 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822123 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-systemd\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.822221 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822149 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-slash\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822221 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822182 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-dbus\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.822221 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822207 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-etc-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822403 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822241 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-node-log\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822403 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822264 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822403 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822288 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.822403 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822311 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22gp\" (UniqueName: \"kubernetes.io/projected/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-kube-api-access-q22gp\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.822403 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822346 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysctl-conf\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.822403 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822367 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-systemd\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822403 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822388 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-kubelet\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822441 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysconfig\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822473 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/004c774c-b87b-4f93-a276-2cab75c42172-etc-tuned\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822498 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-ovn\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822521 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822555 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-ovnkube-script-lib\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822579 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-etc-selinux\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822613 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-cni-bin\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.822697 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822651 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6845d82a-8317-441b-b59a-41bc1b86c9df-host-slash\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822703 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6845d82a-8317-441b-b59a-41bc1b86c9df-host-slash\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822724 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-cni-bin\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822741 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-cni-netd\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822792 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qplf4\" (UniqueName: \"kubernetes.io/projected/db8e86ee-d321-4fdc-be9a-495560a2f399-kube-api-access-qplf4\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822816 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-k8s-cni-cncf-io\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822847 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtp7\" (UniqueName: \"kubernetes.io/projected/b01aab23-af40-42ef-914f-2ca0ea129769-kube-api-access-swtp7\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822869 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-run\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822890 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-host\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822911 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-system-cni-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822933 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-cni-multus\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.822957 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlsg\" (UniqueName: \"kubernetes.io/projected/004c774c-b87b-4f93-a276-2cab75c42172-kube-api-access-szlsg\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.823003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823001 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-run-netns\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823023 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-var-lib-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823047 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-kubelet-config\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823074 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823097 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-cni-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823132 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-cnibin\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823132 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-system-cni-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823164 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-etc-kubernetes\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823190 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/637ed0ce-8721-46e2-8c84-bd306db605c0-agent-certs\") pod \"konnectivity-agent-ql89z\" (UID: \"637ed0ce-8721-46e2-8c84-bd306db605c0\") " pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823193 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-k8s-cni-cncf-io\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823166 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-cni-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823236 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-cni-multus\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823253 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-etc-kubernetes\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823283 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823308 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823332 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-socket-dir-parent\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823393 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823421 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-cnibin\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.823461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823444 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-socket-dir-parent\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823483 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-kubelet\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823550 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-var-lib-kubelet\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823569 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6845d82a-8317-441b-b59a-41bc1b86c9df-iptables-alerter-script\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823597 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-os-release\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823620 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b01aab23-af40-42ef-914f-2ca0ea129769-cni-binary-copy\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823645 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jmr\" (UniqueName: \"kubernetes.io/projected/c989fdb9-e253-413c-b23f-8d6c1389d16c-kube-api-access-n2jmr\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823670 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/004c774c-b87b-4f93-a276-2cab75c42172-tmp\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823661 2545 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823700 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-systemd-units\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823725 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-log-socket\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823748 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-cni-bin\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823767 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-registration-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823789 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823814 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-system-cni-dir\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823839 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-hostroot\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823868 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-multus-certs\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.824306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823896 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-lib-modules\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823922 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.823944 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824002 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhp5\" (UniqueName: \"kubernetes.io/projected/452be7ad-2d9a-4448-899a-d5e68cc10f2e-kube-api-access-tmhp5\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824030 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b01aab23-af40-42ef-914f-2ca0ea129769-multus-daemon-config\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824058 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c989fdb9-e253-413c-b23f-8d6c1389d16c-hosts-file\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824082 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-modprobe-d\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824103 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysctl-d\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824125 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-device-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824149 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64be67c0-7c44-4334-b8d7-29aa305f5fd5-host\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824175 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-netns\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824197 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c989fdb9-e253-413c-b23f-8d6c1389d16c-tmp-dir\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824204 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6845d82a-8317-441b-b59a-41bc1b86c9df-iptables-alerter-script\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824218 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-sys\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824257 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824205 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b01aab23-af40-42ef-914f-2ca0ea129769-cni-binary-copy\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824271 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-os-release\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.825069 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824278 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-sys-fs\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824313 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbgn\" (UniqueName: \"kubernetes.io/projected/64be67c0-7c44-4334-b8d7-29aa305f5fd5-kube-api-access-llbgn\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824358 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-hostroot\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824370 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-netns\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824403 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/637ed0ce-8721-46e2-8c84-bd306db605c0-konnectivity-ca\") pod \"konnectivity-agent-ql89z\" (UID: \"637ed0ce-8721-46e2-8c84-bd306db605c0\") " pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824415 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-cni-binary-copy\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824425 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-host-run-multus-certs\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.824438 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824482 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-system-cni-dir\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.824505 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:45:01.324487222 +0000 UTC m=+2.973419767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824525 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-cnibin\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824553 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-os-release\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824577 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-ovnkube-config\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824602 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-env-overrides\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824639 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91988be0-a23c-4f97-a30f-200ceb718da0-ovn-node-metrics-cert\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824698 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbrk\" (UniqueName: \"kubernetes.io/projected/91988be0-a23c-4f97-a30f-200ceb718da0-kube-api-access-ffbrk\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824736 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db8e86ee-d321-4fdc-be9a-495560a2f399-cnibin\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.825790 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824748 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64be67c0-7c44-4334-b8d7-29aa305f5fd5-serviceca\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824704 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-os-release\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824803 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824857 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-conf-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824890 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b01aab23-af40-42ef-914f-2ca0ea129769-multus-conf-dir\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824895 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgrk\" (UniqueName: \"kubernetes.io/projected/6845d82a-8317-441b-b59a-41bc1b86c9df-kube-api-access-ftgrk\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824942 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-var-lib-kubelet\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824984 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-socket-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.824987 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b01aab23-af40-42ef-914f-2ca0ea129769-multus-daemon-config\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.825085 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/637ed0ce-8721-46e2-8c84-bd306db605c0-konnectivity-ca\") pod \"konnectivity-agent-ql89z\" (UID: \"637ed0ce-8721-46e2-8c84-bd306db605c0\") " pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.825357 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.826470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.825529 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db8e86ee-d321-4fdc-be9a-495560a2f399-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.827470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.827450 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/637ed0ce-8721-46e2-8c84-bd306db605c0-agent-certs\") pod \"konnectivity-agent-ql89z\" (UID: \"637ed0ce-8721-46e2-8c84-bd306db605c0\") " pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:00.831729 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.831710 2545 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 10 10:45:00.832260 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.832227 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtp7\" (UniqueName: \"kubernetes.io/projected/b01aab23-af40-42ef-914f-2ca0ea129769-kube-api-access-swtp7\") pod \"multus-h4kch\" (UID: \"b01aab23-af40-42ef-914f-2ca0ea129769\") " pod="openshift-multus/multus-h4kch" Jun 10 10:45:00.833167 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.833144 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qplf4\" (UniqueName: \"kubernetes.io/projected/db8e86ee-d321-4fdc-be9a-495560a2f399-kube-api-access-qplf4\") pod \"multus-additional-cni-plugins-cxkhd\" (UID: \"db8e86ee-d321-4fdc-be9a-495560a2f399\") " pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:00.838014 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.837948 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgrk\" (UniqueName: \"kubernetes.io/projected/6845d82a-8317-441b-b59a-41bc1b86c9df-kube-api-access-ftgrk\") pod \"iptables-alerter-498lq\" (UID: \"6845d82a-8317-441b-b59a-41bc1b86c9df\") " pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:00.838542 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.838521 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhp5\" (UniqueName: \"kubernetes.io/projected/452be7ad-2d9a-4448-899a-d5e68cc10f2e-kube-api-access-tmhp5\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:00.925398 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925366 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jmr\" (UniqueName: \"kubernetes.io/projected/c989fdb9-e253-413c-b23f-8d6c1389d16c-kube-api-access-n2jmr\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.925527 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925407 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/004c774c-b87b-4f93-a276-2cab75c42172-tmp\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.925527 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925433 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-systemd-units\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.925635 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925559 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-log-socket\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.925635 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925596 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-cni-bin\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.925635 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925607 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-systemd-units\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.925635 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925621 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-registration-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925669 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-registration-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925679 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-lib-modules\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925705 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925707 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-log-socket\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925726 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925732 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-cni-bin\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925755 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c989fdb9-e253-413c-b23f-8d6c1389d16c-hosts-file\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925778 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-modprobe-d\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925782 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-lib-modules\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.925813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925805 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysctl-d\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925824 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-device-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.925832 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925848 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64be67c0-7c44-4334-b8d7-29aa305f5fd5-host\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925854 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-modprobe-d\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925854 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c989fdb9-e253-413c-b23f-8d6c1389d16c-hosts-file\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925878 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c989fdb9-e253-413c-b23f-8d6c1389d16c-tmp-dir\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.925890 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret podName:ff371f6a-32fb-42d5-96f8-d1f6558bf6fa nodeName:}" failed. No retries permitted until 2026-06-10 10:45:01.425871798 +0000 UTC m=+3.074804320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret") pod "global-pull-secret-syncer-47m9g" (UID: "ff371f6a-32fb-42d5-96f8-d1f6558bf6fa") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925912 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64be67c0-7c44-4334-b8d7-29aa305f5fd5-host\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925924 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-sys\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925948 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysctl-d\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925955 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-device-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.925951 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926012 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-sys-fs\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926039 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbgn\" (UniqueName: \"kubernetes.io/projected/64be67c0-7c44-4334-b8d7-29aa305f5fd5-kube-api-access-llbgn\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926014 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-sys\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926067 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-ovnkube-config\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.926287 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926041 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926091 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-env-overrides\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926095 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-sys-fs\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926113 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91988be0-a23c-4f97-a30f-200ceb718da0-ovn-node-metrics-cert\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926135 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbrk\" (UniqueName: \"kubernetes.io/projected/91988be0-a23c-4f97-a30f-200ceb718da0-kube-api-access-ffbrk\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926156 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64be67c0-7c44-4334-b8d7-29aa305f5fd5-serviceca\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926173 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c989fdb9-e253-413c-b23f-8d6c1389d16c-tmp-dir\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926180 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-var-lib-kubelet\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926203 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-socket-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926226 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-kubernetes\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926248 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-systemd\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926271 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-slash\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926286 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-var-lib-kubelet\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926292 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-dbus\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926325 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-etc-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926350 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-node-log\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926372 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927094 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926399 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926403 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-dbus\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926441 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-etc-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926461 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-kubernetes\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926482 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-node-log\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926510 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-systemd\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926519 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926546 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-slash\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926548 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22gp\" (UniqueName: \"kubernetes.io/projected/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-kube-api-access-q22gp\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926585 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysctl-conf\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926607 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-systemd\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926632 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-kubelet\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926644 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-ovnkube-config\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926655 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysconfig\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926676 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/004c774c-b87b-4f93-a276-2cab75c42172-etc-tuned\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926699 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-ovn\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926722 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.927867 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926726 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926743 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-kubelet\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926761 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926767 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-ovnkube-script-lib\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926794 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-etc-selinux\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926808 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysconfig\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926808 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/64be67c0-7c44-4334-b8d7-29aa305f5fd5-serviceca\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926810 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-systemd\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926820 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-cni-netd\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926849 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-cni-netd\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926841 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-run-ovn\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926875 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-run\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926895 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-host\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926918 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-etc-selinux\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926925 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlsg\" (UniqueName: \"kubernetes.io/projected/004c774c-b87b-4f93-a276-2cab75c42172-kube-api-access-szlsg\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926950 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-run-netns\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926980 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-run\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926981 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-etc-sysctl-conf\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.928665 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.926992 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-var-lib-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927025 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-kubelet-config\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927033 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-var-lib-openvswitch\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927042 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91988be0-a23c-4f97-a30f-200ceb718da0-host-run-netns\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927078 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-kubelet-config\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927080 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/004c774c-b87b-4f93-a276-2cab75c42172-host\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927270 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-ovnkube-script-lib\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927272 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-socket-dir\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.927499 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/91988be0-a23c-4f97-a30f-200ceb718da0-env-overrides\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.928063 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/004c774c-b87b-4f93-a276-2cab75c42172-tmp\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.929085 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/91988be0-a23c-4f97-a30f-200ceb718da0-ovn-node-metrics-cert\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:00.929369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.929303 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/004c774c-b87b-4f93-a276-2cab75c42172-etc-tuned\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.940071 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.940026 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:45:00.940071 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.940047 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:45:00.940071 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.940060 2545 projected.go:196] Error preparing data for projected volume kube-api-access-6d8h9 for pod openshift-network-diagnostics/network-check-target-gdqfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:00.940295 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:00.940195 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9 podName:89fd8558-befa-433f-9575-27edb9ebdf62 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:01.440176628 +0000 UTC m=+3.089109177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6d8h9" (UniqueName: "kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9") pod "network-check-target-gdqfd" (UID: "89fd8558-befa-433f-9575-27edb9ebdf62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:00.942211 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.942188 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbgn\" (UniqueName: \"kubernetes.io/projected/64be67c0-7c44-4334-b8d7-29aa305f5fd5-kube-api-access-llbgn\") pod \"node-ca-jgp8h\" (UID: \"64be67c0-7c44-4334-b8d7-29aa305f5fd5\") " pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:00.942353 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.942329 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jmr\" (UniqueName: \"kubernetes.io/projected/c989fdb9-e253-413c-b23f-8d6c1389d16c-kube-api-access-n2jmr\") pod \"node-resolver-p4l5s\" (UID: \"c989fdb9-e253-413c-b23f-8d6c1389d16c\") " pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:00.943059 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.943023 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlsg\" (UniqueName: \"kubernetes.io/projected/004c774c-b87b-4f93-a276-2cab75c42172-kube-api-access-szlsg\") pod \"tuned-p82p8\" (UID: \"004c774c-b87b-4f93-a276-2cab75c42172\") " pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:00.943540 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.943514 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22gp\" (UniqueName: \"kubernetes.io/projected/ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4-kube-api-access-q22gp\") pod \"aws-ebs-csi-driver-node-d7rnc\" (UID: \"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:00.943726 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:00.943708 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbrk\" (UniqueName: \"kubernetes.io/projected/91988be0-a23c-4f97-a30f-200ceb718da0-kube-api-access-ffbrk\") pod \"ovnkube-node-pfkh8\" (UID: \"91988be0-a23c-4f97-a30f-200ceb718da0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:01.011940 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.011910 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:01.021353 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.021325 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" Jun 10 10:45:01.032987 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.032954 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4kch" Jun 10 10:45:01.039570 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.039552 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-498lq" Jun 10 10:45:01.047137 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.047119 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4l5s" Jun 10 10:45:01.052758 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.052740 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" Jun 10 10:45:01.060304 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.060287 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p82p8" Jun 10 10:45:01.066842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.066827 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:01.073360 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.073336 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jgp8h" Jun 10 10:45:01.329869 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.329803 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:01.330023 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.329974 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:01.330064 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.330040 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:45:02.330020218 +0000 UTC m=+3.978952739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:01.430811 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.430768 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:01.430940 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.430895 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:01.431009 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.430955 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret podName:ff371f6a-32fb-42d5-96f8-d1f6558bf6fa nodeName:}" failed. No retries permitted until 2026-06-10 10:45:02.43094159 +0000 UTC m=+4.079874114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret") pod "global-pull-secret-syncer-47m9g" (UID: "ff371f6a-32fb-42d5-96f8-d1f6558bf6fa") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:01.531066 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.531044 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:01.531174 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.531162 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:45:01.531223 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.531179 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:45:01.531223 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.531188 2545 projected.go:196] Error preparing data for projected volume kube-api-access-6d8h9 for pod openshift-network-diagnostics/network-check-target-gdqfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:01.531292 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:01.531230 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9 podName:89fd8558-befa-433f-9575-27edb9ebdf62 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:02.53121767 +0000 UTC m=+4.180150189 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d8h9" (UniqueName: "kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9") pod "network-check-target-gdqfd" (UID: "89fd8558-befa-433f-9575-27edb9ebdf62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:01.538542 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.538513 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91988be0_a23c_4f97_a30f_200ceb718da0.slice/crio-580cd1460be477f1d206f8561222dd0d01e88b8fa0eb0c8e34f23868c833d0cd WatchSource:0}: Error finding container 580cd1460be477f1d206f8561222dd0d01e88b8fa0eb0c8e34f23868c833d0cd: Status 404 returned error can't find the container with id 580cd1460be477f1d206f8561222dd0d01e88b8fa0eb0c8e34f23868c833d0cd Jun 10 10:45:01.539566 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.539542 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637ed0ce_8721_46e2_8c84_bd306db605c0.slice/crio-5f99f5332e531dcd10f49d5a7df26015bee60d9d4a47e3758464eedff09f66f7 WatchSource:0}: Error finding container 5f99f5332e531dcd10f49d5a7df26015bee60d9d4a47e3758464eedff09f66f7: Status 404 returned error can't find the container with id 5f99f5332e531dcd10f49d5a7df26015bee60d9d4a47e3758464eedff09f66f7 Jun 10 10:45:01.561557 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.561534 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8e86ee_d321_4fdc_be9a_495560a2f399.slice/crio-60cba152bde43f2befe374be4b67c36764fb02249d703c73b800b23a69e0e32c WatchSource:0}: Error finding container 60cba152bde43f2befe374be4b67c36764fb02249d703c73b800b23a69e0e32c: Status 404 returned error can't find the container with id 60cba152bde43f2befe374be4b67c36764fb02249d703c73b800b23a69e0e32c Jun 10 10:45:01.564297 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.564275 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade755c1_7cc2_40d5_8a7b_d29f8ec3fec4.slice/crio-88e7ce8537e3656031da7db4a753ae93d1e44c36c9512507af76c6c5f955f139 WatchSource:0}: Error finding container 88e7ce8537e3656031da7db4a753ae93d1e44c36c9512507af76c6c5f955f139: Status 404 returned error can't find the container with id 88e7ce8537e3656031da7db4a753ae93d1e44c36c9512507af76c6c5f955f139 Jun 10 10:45:01.564645 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.564619 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc989fdb9_e253_413c_b23f_8d6c1389d16c.slice/crio-89f42e5f8c312a0f0f48affaff5291ea08d198644382cf8f415bc5ebbb8775ca WatchSource:0}: Error finding container 89f42e5f8c312a0f0f48affaff5291ea08d198644382cf8f415bc5ebbb8775ca: Status 404 returned error can't find the container with id 89f42e5f8c312a0f0f48affaff5291ea08d198644382cf8f415bc5ebbb8775ca Jun 10 10:45:01.565857 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.565842 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64be67c0_7c44_4334_b8d7_29aa305f5fd5.slice/crio-5b8bbfa0fca65fbe5155a71a627124bec3d3024ddaad86b6f96e8e4f4d168ec3 WatchSource:0}: Error finding container 5b8bbfa0fca65fbe5155a71a627124bec3d3024ddaad86b6f96e8e4f4d168ec3: Status 404 returned error can't find the container with id 5b8bbfa0fca65fbe5155a71a627124bec3d3024ddaad86b6f96e8e4f4d168ec3 Jun 10 10:45:01.566980 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.566946 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6845d82a_8317_441b_b59a_41bc1b86c9df.slice/crio-b4ee6b5ebb5df4da3d9b9e4e0f05535c1f1a3c90228c331c35a164f6eb0111c2 WatchSource:0}: Error finding container b4ee6b5ebb5df4da3d9b9e4e0f05535c1f1a3c90228c331c35a164f6eb0111c2: Status 404 returned error can't find the container with id b4ee6b5ebb5df4da3d9b9e4e0f05535c1f1a3c90228c331c35a164f6eb0111c2 Jun 10 10:45:01.568500 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.568478 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01aab23_af40_42ef_914f_2ca0ea129769.slice/crio-773bcde6891a6b754a12d64066fc5488ca264315dc8dd9128a43a1e36cd53df4 WatchSource:0}: Error finding container 773bcde6891a6b754a12d64066fc5488ca264315dc8dd9128a43a1e36cd53df4: Status 404 returned error can't find the container with id 773bcde6891a6b754a12d64066fc5488ca264315dc8dd9128a43a1e36cd53df4 Jun 10 10:45:01.569082 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:01.569061 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004c774c_b87b_4f93_a276_2cab75c42172.slice/crio-719a6a61160a67992ddee1eb36900abc9182b797ecd827e9ff219bf8a3350766 WatchSource:0}: Error finding container 719a6a61160a67992ddee1eb36900abc9182b797ecd827e9ff219bf8a3350766: Status 404 returned error can't find the container with id 719a6a61160a67992ddee1eb36900abc9182b797ecd827e9ff219bf8a3350766 Jun 10 10:45:01.751984 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.751918 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-09 10:40:00 +0000 UTC" deadline="2028-01-15 15:54:34.994714158 +0000 UTC" Jun 10 10:45:01.751984 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.751980 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14021h9m33.242755006s" Jun 10 10:45:01.785580 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.785549 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4l5s" event={"ID":"c989fdb9-e253-413c-b23f-8d6c1389d16c","Type":"ContainerStarted","Data":"89f42e5f8c312a0f0f48affaff5291ea08d198644382cf8f415bc5ebbb8775ca"} Jun 10 10:45:01.786714 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.786643 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" event={"ID":"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4","Type":"ContainerStarted","Data":"88e7ce8537e3656031da7db4a753ae93d1e44c36c9512507af76c6c5f955f139"} Jun 10 10:45:01.787739 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.787720 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerStarted","Data":"60cba152bde43f2befe374be4b67c36764fb02249d703c73b800b23a69e0e32c"} Jun 10 10:45:01.788794 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.788772 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ql89z" event={"ID":"637ed0ce-8721-46e2-8c84-bd306db605c0","Type":"ContainerStarted","Data":"5f99f5332e531dcd10f49d5a7df26015bee60d9d4a47e3758464eedff09f66f7"} Jun 10 10:45:01.790126 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.790106 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"580cd1460be477f1d206f8561222dd0d01e88b8fa0eb0c8e34f23868c833d0cd"} Jun 10 10:45:01.791834 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.791813 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" event={"ID":"75bf6497d56e225e5bd01fc8d3fdd351","Type":"ContainerStarted","Data":"aeced862db6085146757f9529bf098da852b1d226f588ae25515e43fca5b6b88"} Jun 10 10:45:01.793824 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.793803 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p82p8" event={"ID":"004c774c-b87b-4f93-a276-2cab75c42172","Type":"ContainerStarted","Data":"719a6a61160a67992ddee1eb36900abc9182b797ecd827e9ff219bf8a3350766"} Jun 10 10:45:01.796139 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.796104 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4kch" event={"ID":"b01aab23-af40-42ef-914f-2ca0ea129769","Type":"ContainerStarted","Data":"773bcde6891a6b754a12d64066fc5488ca264315dc8dd9128a43a1e36cd53df4"} Jun 10 10:45:01.797171 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.797155 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-498lq" event={"ID":"6845d82a-8317-441b-b59a-41bc1b86c9df","Type":"ContainerStarted","Data":"b4ee6b5ebb5df4da3d9b9e4e0f05535c1f1a3c90228c331c35a164f6eb0111c2"} Jun 10 10:45:01.798126 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.798107 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jgp8h" event={"ID":"64be67c0-7c44-4334-b8d7-29aa305f5fd5","Type":"ContainerStarted","Data":"5b8bbfa0fca65fbe5155a71a627124bec3d3024ddaad86b6f96e8e4f4d168ec3"} Jun 10 10:45:01.805997 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:01.805932 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" podStartSLOduration=2.805916987 podStartE2EDuration="2.805916987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:01.805373834 +0000 UTC m=+3.454306376" watchObservedRunningTime="2026-06-10 10:45:01.805916987 +0000 UTC m=+3.454849527" Jun 10 10:45:02.338786 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.338748 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:02.338989 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.338914 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:02.339064 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.338993 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:45:04.338958729 +0000 UTC m=+5.987891273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:02.439849 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.439811 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:02.440084 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.439999 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:02.440084 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.440062 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret podName:ff371f6a-32fb-42d5-96f8-d1f6558bf6fa nodeName:}" failed. No retries permitted until 2026-06-10 10:45:04.440043 +0000 UTC m=+6.088975543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret") pod "global-pull-secret-syncer-47m9g" (UID: "ff371f6a-32fb-42d5-96f8-d1f6558bf6fa") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:02.540550 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.540514 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:02.540769 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.540750 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:45:02.540847 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.540779 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:45:02.540847 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.540792 2545 projected.go:196] Error preparing data for projected volume kube-api-access-6d8h9 for pod openshift-network-diagnostics/network-check-target-gdqfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:02.541039 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.540850 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9 podName:89fd8558-befa-433f-9575-27edb9ebdf62 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:04.540831895 +0000 UTC m=+6.189764432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d8h9" (UniqueName: "kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9") pod "network-check-target-gdqfd" (UID: "89fd8558-befa-433f-9575-27edb9ebdf62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:02.752598 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.752449 2545 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-09 10:40:00 +0000 UTC" deadline="2028-02-21 05:13:30.71431143 +0000 UTC" Jun 10 10:45:02.752598 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.752486 2545 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14898h28m27.961830635s" Jun 10 10:45:02.779186 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.779145 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:02.779347 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.779278 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:02.780598 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.779661 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:02.780598 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.779747 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:02.780598 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.779817 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:02.780598 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:02.779887 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:02.816278 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.816234 2545 generic.go:356] "Generic (PLEG): container finished" podID="29e8a60c6729dc0922027f4cd7063603" containerID="cc0e6a941454c04ebc9a23fa1eacbdfee4444e0eb88c2522f66d5fe0132c4d6b" exitCode=0 Jun 10 10:45:02.816413 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:02.816330 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" event={"ID":"29e8a60c6729dc0922027f4cd7063603","Type":"ContainerDied","Data":"cc0e6a941454c04ebc9a23fa1eacbdfee4444e0eb88c2522f66d5fe0132c4d6b"} Jun 10 10:45:03.827984 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:03.827931 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" event={"ID":"29e8a60c6729dc0922027f4cd7063603","Type":"ContainerStarted","Data":"6646b7f80a0202c8900e94d1113aeec53670604f150fb7565ab49b8e6151a10e"} Jun 10 10:45:03.846037 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:03.845986 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" podStartSLOduration=4.84595301 podStartE2EDuration="4.84595301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:03.845542569 +0000 UTC m=+5.494475111" watchObservedRunningTime="2026-06-10 10:45:03.84595301 +0000 UTC m=+5.494885552" Jun 10 10:45:04.356583 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:04.355924 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:04.356583 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.356133 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:04.356583 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.356204 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:45:08.356183286 +0000 UTC m=+10.005115808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:04.456613 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:04.456568 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:04.456807 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.456722 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:04.456807 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.456783 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret podName:ff371f6a-32fb-42d5-96f8-d1f6558bf6fa nodeName:}" failed. No retries permitted until 2026-06-10 10:45:08.456766309 +0000 UTC m=+10.105698852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret") pod "global-pull-secret-syncer-47m9g" (UID: "ff371f6a-32fb-42d5-96f8-d1f6558bf6fa") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:04.557984 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:04.557935 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:04.558152 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.558118 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:45:04.558152 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.558143 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:45:04.558260 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.558156 2545 projected.go:196] Error preparing data for projected volume kube-api-access-6d8h9 for pod openshift-network-diagnostics/network-check-target-gdqfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:04.558260 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.558236 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9 podName:89fd8558-befa-433f-9575-27edb9ebdf62 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:08.558201857 +0000 UTC m=+10.207134377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d8h9" (UniqueName: "kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9") pod "network-check-target-gdqfd" (UID: "89fd8558-befa-433f-9575-27edb9ebdf62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:04.777428 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:04.777355 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:04.777585 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.777508 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:04.777983 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:04.777871 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:04.777983 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:04.777836 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:04.778157 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.778009 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:04.778157 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:04.778087 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:06.777084 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:06.777047 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:06.777576 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:06.777049 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:06.777576 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:06.777209 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:06.777576 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:06.777213 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:06.777576 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:06.777333 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:06.777576 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:06.777411 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:08.389185 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:08.389088 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:08.389661 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.389314 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:08.389661 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.389384 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:45:16.389369806 +0000 UTC m=+18.038302325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:08.490583 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:08.490275 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:08.490583 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.490468 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:08.490583 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.490529 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret podName:ff371f6a-32fb-42d5-96f8-d1f6558bf6fa nodeName:}" failed. No retries permitted until 2026-06-10 10:45:16.490509906 +0000 UTC m=+18.139442444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret") pod "global-pull-secret-syncer-47m9g" (UID: "ff371f6a-32fb-42d5-96f8-d1f6558bf6fa") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:08.590690 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:08.590652 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:08.590847 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.590821 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:45:08.590847 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.590842 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:45:08.590989 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.590854 2545 projected.go:196] Error preparing data for projected volume kube-api-access-6d8h9 for pod openshift-network-diagnostics/network-check-target-gdqfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:08.590989 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.590909 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9 podName:89fd8558-befa-433f-9575-27edb9ebdf62 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:16.590890576 +0000 UTC m=+18.239823115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d8h9" (UniqueName: "kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9") pod "network-check-target-gdqfd" (UID: "89fd8558-befa-433f-9575-27edb9ebdf62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:08.777750 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:08.777236 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:08.777750 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.777354 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:08.777750 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:08.777418 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:08.777750 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.777532 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:08.777750 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:08.777557 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:08.777750 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:08.777650 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:10.780183 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:10.780111 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:10.780183 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:10.780158 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:10.780183 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:10.780158 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:10.780679 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:10.780241 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:10.780679 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:10.780357 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:10.780679 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:10.780471 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:12.776726 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:12.776693 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:12.776726 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:12.776704 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:12.776726 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:12.776700 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:12.777265 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:12.776824 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:12.777265 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:12.776913 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:12.777265 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:12.777019 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:14.777378 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:14.777345 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:14.777779 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:14.777346 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:14.777779 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:14.777459 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:14.777779 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:14.777351 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:14.777779 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:14.777536 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:14.777779 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:14.777637 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:16.454519 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:16.454484 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:16.454944 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.454661 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:16.454944 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.454737 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:45:32.454714392 +0000 UTC m=+34.103646916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:16.555165 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:16.555089 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:16.555316 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.555221 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:16.555316 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.555285 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret podName:ff371f6a-32fb-42d5-96f8-d1f6558bf6fa nodeName:}" failed. No retries permitted until 2026-06-10 10:45:32.555267059 +0000 UTC m=+34.204199579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret") pod "global-pull-secret-syncer-47m9g" (UID: "ff371f6a-32fb-42d5-96f8-d1f6558bf6fa") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:16.655806 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:16.655777 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:16.656074 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.655975 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:45:16.656074 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.655998 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:45:16.656074 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.656013 2545 projected.go:196] Error preparing data for projected volume kube-api-access-6d8h9 for pod openshift-network-diagnostics/network-check-target-gdqfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:16.656074 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.656072 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9 podName:89fd8558-befa-433f-9575-27edb9ebdf62 nodeName:}" failed. No retries permitted until 2026-06-10 10:45:32.656056837 +0000 UTC m=+34.304989379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d8h9" (UniqueName: "kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9") pod "network-check-target-gdqfd" (UID: "89fd8558-befa-433f-9575-27edb9ebdf62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:16.776975 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:16.776930 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:16.776975 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:16.776959 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:16.777181 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:16.776942 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:16.777181 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.777076 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:16.777181 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.777142 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:16.777318 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:16.777220 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:18.778178 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:18.778125 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:18.778629 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:18.778231 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:18.778629 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:18.778243 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:18.778629 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:18.778339 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:18.778629 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:18.778345 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:18.778629 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:18.778419 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:20.776912 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.776589 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:20.777669 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.776594 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:20.777669 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:20.777010 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:20.777669 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:20.777065 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:20.777669 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.776620 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:20.777669 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:20.777158 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:20.861219 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.861192 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jgp8h" event={"ID":"64be67c0-7c44-4334-b8d7-29aa305f5fd5","Type":"ContainerStarted","Data":"3d7615b878ccf6bb4926f6f8dfce49e8b6e8cc1033afd545dc628bcbea06ed5e"} Jun 10 10:45:20.862873 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.862847 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4l5s" event={"ID":"c989fdb9-e253-413c-b23f-8d6c1389d16c","Type":"ContainerStarted","Data":"9e1009638c0065e5820f4808118ff1374ca04dd66565e4fc04340f294646d7f2"} Jun 10 10:45:20.864536 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.864513 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" event={"ID":"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4","Type":"ContainerStarted","Data":"c6babd3a21b84bf46236f8ddedba4c7605fc20668a5a710a218238e886b81115"} Jun 10 10:45:20.866526 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.866466 2545 generic.go:356] "Generic (PLEG): container finished" podID="db8e86ee-d321-4fdc-be9a-495560a2f399" containerID="583ec9b501f899d02df84b3ff7375b51a2690a4aaece5eb24b20eaf593698911" exitCode=0 Jun 10 10:45:20.866615 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.866548 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerDied","Data":"583ec9b501f899d02df84b3ff7375b51a2690a4aaece5eb24b20eaf593698911"} Jun 10 10:45:20.868021 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.867998 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ql89z" event={"ID":"637ed0ce-8721-46e2-8c84-bd306db605c0","Type":"ContainerStarted","Data":"69d593fa11e8a38ebd1aba37b6ccab4876de910a104040d0274f9a8ea977e104"} Jun 10 10:45:20.871549 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.871523 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"4b76057de05c8d7702898604cb0bae1c2fd31e92c0181f6b382aeb787ed9c43c"} Jun 10 10:45:20.871646 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.871604 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"95ea4afb24bdd900b1d3944486ee744fe9081314908803db086a5bffe5876c7a"} Jun 10 10:45:20.871695 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.871658 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"de509db14673a3bdcd686d133f10b2157849580bbdde635f39e7f09a11173a99"} Jun 10 10:45:20.871746 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.871704 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"4ee011c127a243ef948796b49f61246e61690554af5bdfb7842a1c2310ea5c00"} Jun 10 10:45:20.871797 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.871756 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"f3514b56ca9aa2701b3b7eef99cc433e965fb343b254e83398a9388cb855d684"} Jun 10 10:45:20.871847 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.871799 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"3083ba57ef7c3e2c42a1db7d634d2313a1cecf385f8be764dc4f05eb46d3d700"} Jun 10 10:45:20.875415 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.875201 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p82p8" event={"ID":"004c774c-b87b-4f93-a276-2cab75c42172","Type":"ContainerStarted","Data":"6e23a4a7a96e18fad2f0ec7b74564fa4b8234734b9a8b1e0ad7269b71d9cba00"} Jun 10 10:45:20.877241 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.877219 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4kch" event={"ID":"b01aab23-af40-42ef-914f-2ca0ea129769","Type":"ContainerStarted","Data":"46d3551e4b14a1a0c6629cd7906358c214a683c725d10be1b92f3ec42f054d17"} Jun 10 10:45:20.877356 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.877304 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jgp8h" podStartSLOduration=3.280119386 podStartE2EDuration="21.877272686s" totalImagesPullingTime="18.5971533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:20.877252606 +0000 UTC m=+22.526185148" watchObservedRunningTime="2026-06-10 10:45:20.877272686 +0000 UTC m=+22.526205226" Jun 10 10:45:20.892125 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.892080 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p4l5s" podStartSLOduration=4.293524104 podStartE2EDuration="22.892063494s" totalImagesPullingTime="18.59853939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:20.8911806 +0000 UTC m=+22.540113143" watchObservedRunningTime="2026-06-10 10:45:20.892063494 +0000 UTC m=+22.540996035" Jun 10 10:45:20.910301 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.910254 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p82p8" podStartSLOduration=3.303952759 podStartE2EDuration="21.910241965s" totalImagesPullingTime="18.606289206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:20.910210045 +0000 UTC m=+22.559142586" watchObservedRunningTime="2026-06-10 10:45:20.910241965 +0000 UTC m=+22.559174505" Jun 10 10:45:20.948003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.947951 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ql89z" podStartSLOduration=12.282715436 podStartE2EDuration="22.947935126s" totalImagesPullingTime="10.66521969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:20.947449303 +0000 UTC m=+22.596381843" watchObservedRunningTime="2026-06-10 10:45:20.947935126 +0000 UTC m=+22.596867668" Jun 10 10:45:20.970659 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:20.970617 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h4kch" podStartSLOduration=4.322165292 podStartE2EDuration="22.970604014s" totalImagesPullingTime="18.648438722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:20.970094313 +0000 UTC m=+22.619026855" watchObservedRunningTime="2026-06-10 10:45:20.970604014 +0000 UTC m=+22.619536554" Jun 10 10:45:21.559657 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:21.559631 2545 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jun 10 10:45:21.777254 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:21.777098 2545 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-10T10:45:21.55965375Z","UUID":"32d57f00-f45b-4b56-8ef0-289fb9687a38","Handler":null,"Name":"","Endpoint":""} Jun 10 10:45:21.778620 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:21.778599 2545 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jun 10 10:45:21.778737 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:21.778643 2545 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jun 10 10:45:21.881377 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:21.881338 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" event={"ID":"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4","Type":"ContainerStarted","Data":"2bfdc30f71fcef91814cb4e78390f3fa66a048e0f8945bea23d0e76368f56861"} Jun 10 10:45:21.883022 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:21.882987 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-498lq" event={"ID":"6845d82a-8317-441b-b59a-41bc1b86c9df","Type":"ContainerStarted","Data":"c1d70627753b07d3c69a639b0e2169e9e49eb924aa3a7b0530675c1f8313c90c"} Jun 10 10:45:21.900431 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:21.900386 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-498lq" podStartSLOduration=5.3033139590000005 podStartE2EDuration="23.900369921s" totalImagesPullingTime="18.597055962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:21.900021436 +0000 UTC m=+23.548953979" watchObservedRunningTime="2026-06-10 10:45:21.900369921 +0000 UTC m=+23.549302461" Jun 10 10:45:22.777203 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:22.777166 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:22.777203 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:22.777206 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:22.777733 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:22.777168 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:22.777733 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:22.777294 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:22.777733 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:22.777407 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:22.777733 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:22.777495 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:22.887407 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:22.887369 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" event={"ID":"ade755c1-7cc2-40d5-8a7b-d29f8ec3fec4","Type":"ContainerStarted","Data":"dab8ada3740731367c9ffe523c272871cf2620daa0309e4168de4d8d2047bdb7"} Jun 10 10:45:23.247914 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:23.247837 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:23.248483 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:23.248463 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:23.265931 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:23.265885 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d7rnc" podStartSLOduration=3.626143761 podStartE2EDuration="24.265868698s" totalImagesPullingTime="20.639724937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:22.909728403 +0000 UTC m=+24.558660943" watchObservedRunningTime="2026-06-10 10:45:23.265868698 +0000 UTC m=+24.914801257" Jun 10 10:45:23.894140 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:23.893926 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"d980f4e5c50f52f83a4f80848b4aa301462caed0058838af66d6a867101ff8bc"} Jun 10 10:45:23.894574 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:23.894379 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:23.894758 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:23.894734 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ql89z" Jun 10 10:45:24.776685 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:24.776658 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:24.776859 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:24.776658 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:24.776859 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:24.776773 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:24.776859 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:24.776666 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:24.776859 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:24.776844 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:24.777081 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:24.776947 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:25.899450 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:25.899287 2545 generic.go:356] "Generic (PLEG): container finished" podID="db8e86ee-d321-4fdc-be9a-495560a2f399" containerID="edc3e3c621c9c77fdd0c5fa71a09f049af6a319ddced2d395858704614d87b8d" exitCode=0 Jun 10 10:45:25.900230 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:25.899365 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerDied","Data":"edc3e3c621c9c77fdd0c5fa71a09f049af6a319ddced2d395858704614d87b8d"} Jun 10 10:45:25.903146 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:25.903127 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" event={"ID":"91988be0-a23c-4f97-a30f-200ceb718da0","Type":"ContainerStarted","Data":"50df37a1a55213091a38bd78485b92848a87c76dd12648a80dbb7b15777b8866"} Jun 10 10:45:25.903359 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:25.903346 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:25.903419 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:25.903384 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:25.918651 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:25.918628 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:25.946832 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:25.946793 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" podStartSLOduration=8.251074926 podStartE2EDuration="26.946782869s" totalImagesPullingTime="18.695707943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:25.945397237 +0000 UTC m=+27.594329776" watchObservedRunningTime="2026-06-10 10:45:25.946782869 +0000 UTC m=+27.595715409" Jun 10 10:45:26.776586 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:26.776560 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:26.776789 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:26.776560 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:26.776789 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:26.776657 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:26.776789 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:26.776670 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:26.776885 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:26.776785 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:26.776914 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:26.776884 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:26.905587 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:26.905563 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:26.919216 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:26.919192 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:45:27.058194 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.058166 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-47m9g"] Jun 10 10:45:27.058290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.058270 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:27.058381 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:27.058359 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:27.061353 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.061325 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gdqfd"] Jun 10 10:45:27.061490 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.061432 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:27.061546 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:27.061526 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:27.061840 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.061819 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xj9p"] Jun 10 10:45:27.061917 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.061905 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:27.062044 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:27.062024 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:27.909014 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.908838 2545 generic.go:356] "Generic (PLEG): container finished" podID="db8e86ee-d321-4fdc-be9a-495560a2f399" containerID="84800cfb94882c7f4c1dced884e54f01a912ac4dacc1292122805713b329d3d1" exitCode=0 Jun 10 10:45:27.909530 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:27.908921 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerDied","Data":"84800cfb94882c7f4c1dced884e54f01a912ac4dacc1292122805713b329d3d1"} Jun 10 10:45:28.777765 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:28.777731 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:28.777913 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:28.777834 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:28.777913 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:28.777875 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:28.777913 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:28.777867 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:28.778046 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:28.777955 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:28.778090 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:28.778072 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:29.915629 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:29.915538 2545 generic.go:356] "Generic (PLEG): container finished" podID="db8e86ee-d321-4fdc-be9a-495560a2f399" containerID="ae73e11d491e67825f31bff987b31f44caeaff79c7236b64a8e7206524ae7a57" exitCode=0 Jun 10 10:45:29.915629 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:29.915600 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerDied","Data":"ae73e11d491e67825f31bff987b31f44caeaff79c7236b64a8e7206524ae7a57"} Jun 10 10:45:30.776750 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:30.776688 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:30.776934 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:30.776700 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:30.776934 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:30.776819 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:30.777075 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:30.776979 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:30.777075 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:30.777018 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:30.777193 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:30.777155 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:32.483083 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:32.483052 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:32.483427 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.483150 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:32.483427 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.483197 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:46:04.483184319 +0000 UTC m=+66.132116841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 10 10:45:32.584074 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:32.584045 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:32.584231 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.584213 2545 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:32.584295 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.584277 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret podName:ff371f6a-32fb-42d5-96f8-d1f6558bf6fa nodeName:}" failed. No retries permitted until 2026-06-10 10:46:04.584257375 +0000 UTC m=+66.233189915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret") pod "global-pull-secret-syncer-47m9g" (UID: "ff371f6a-32fb-42d5-96f8-d1f6558bf6fa") : object "kube-system"/"original-pull-secret" not registered Jun 10 10:45:32.685085 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:32.685059 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:32.685227 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.685190 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 10 10:45:32.685227 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.685207 2545 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 10 10:45:32.685227 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.685219 2545 projected.go:196] Error preparing data for projected volume kube-api-access-6d8h9 for pod openshift-network-diagnostics/network-check-target-gdqfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:32.685363 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.685270 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9 podName:89fd8558-befa-433f-9575-27edb9ebdf62 nodeName:}" failed. No retries permitted until 2026-06-10 10:46:04.685254286 +0000 UTC m=+66.334186821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6d8h9" (UniqueName: "kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9") pod "network-check-target-gdqfd" (UID: "89fd8558-befa-433f-9575-27edb9ebdf62") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 10 10:45:32.776917 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:32.776842 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:32.777075 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:32.776846 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:32.777075 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.776978 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdqfd" podUID="89fd8558-befa-433f-9575-27edb9ebdf62" Jun 10 10:45:32.777075 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:32.776995 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:32.777233 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.777122 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-47m9g" podUID="ff371f6a-32fb-42d5-96f8-d1f6558bf6fa" Jun 10 10:45:32.777233 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:32.777212 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:45:33.134984 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.134936 2545 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeReady" Jun 10 10:45:33.135132 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.135100 2545 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jun 10 10:45:33.184683 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.184634 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tc8c9"] Jun 10 10:45:33.187096 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.187073 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.187990 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.187946 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dqngk"] Jun 10 10:45:33.189901 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.189678 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.189901 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.189692 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jun 10 10:45:33.189901 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.189778 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-977dh\"" Jun 10 10:45:33.189901 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.189778 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jun 10 10:45:33.192363 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.192247 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jun 10 10:45:33.192363 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.192283 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jun 10 10:45:33.192363 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.192305 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jun 10 10:45:33.192911 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.192893 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dkm7w\"" Jun 10 10:45:33.198274 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.198250 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tc8c9"] Jun 10 10:45:33.213826 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.213802 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dqngk"] Jun 10 10:45:33.290312 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.290274 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/252cfbef-2c8f-4d25-971d-cf008884454d-config-volume\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.290470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.290322 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.290470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.290350 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2hb\" (UniqueName: \"kubernetes.io/projected/8e608297-9570-4900-bba2-7f7a280d9779-kube-api-access-zw2hb\") pod \"ingress-canary-dqngk\" (UID: \"8e608297-9570-4900-bba2-7f7a280d9779\") " pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.290470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.290380 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252cfbef-2c8f-4d25-971d-cf008884454d-tmp-dir\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.290470 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.290404 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6sc8\" (UniqueName: \"kubernetes.io/projected/252cfbef-2c8f-4d25-971d-cf008884454d-kube-api-access-b6sc8\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.290636 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.290514 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e608297-9570-4900-bba2-7f7a280d9779-cert\") pod \"ingress-canary-dqngk\" (UID: \"8e608297-9570-4900-bba2-7f7a280d9779\") " pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.391299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391223 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/252cfbef-2c8f-4d25-971d-cf008884454d-config-volume\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.391299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391264 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.391299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391284 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2hb\" (UniqueName: \"kubernetes.io/projected/8e608297-9570-4900-bba2-7f7a280d9779-kube-api-access-zw2hb\") pod \"ingress-canary-dqngk\" (UID: \"8e608297-9570-4900-bba2-7f7a280d9779\") " pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.391725 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391311 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252cfbef-2c8f-4d25-971d-cf008884454d-tmp-dir\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.391725 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391338 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6sc8\" (UniqueName: \"kubernetes.io/projected/252cfbef-2c8f-4d25-971d-cf008884454d-kube-api-access-b6sc8\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.391725 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:33.391407 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:33.391725 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:33.391467 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:45:33.891449947 +0000 UTC m=+35.540382484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:45:33.391725 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391407 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e608297-9570-4900-bba2-7f7a280d9779-cert\") pod \"ingress-canary-dqngk\" (UID: \"8e608297-9570-4900-bba2-7f7a280d9779\") " pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.391983 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391827 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/252cfbef-2c8f-4d25-971d-cf008884454d-config-volume\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.391983 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.391866 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252cfbef-2c8f-4d25-971d-cf008884454d-tmp-dir\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.395741 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.395721 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e608297-9570-4900-bba2-7f7a280d9779-cert\") pod \"ingress-canary-dqngk\" (UID: \"8e608297-9570-4900-bba2-7f7a280d9779\") " pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.400262 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.400237 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6sc8\" (UniqueName: \"kubernetes.io/projected/252cfbef-2c8f-4d25-971d-cf008884454d-kube-api-access-b6sc8\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.401056 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.401024 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2hb\" (UniqueName: \"kubernetes.io/projected/8e608297-9570-4900-bba2-7f7a280d9779-kube-api-access-zw2hb\") pod \"ingress-canary-dqngk\" (UID: \"8e608297-9570-4900-bba2-7f7a280d9779\") " pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.510486 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.510454 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dqngk" Jun 10 10:45:33.896303 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:33.896271 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:33.896468 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:33.896452 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:33.896542 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:33.896529 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:45:34.896507781 +0000 UTC m=+36.545440304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:45:34.293401 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.293245 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dqngk"] Jun 10 10:45:34.298511 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:45:34.298486 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e608297_9570_4900_bba2_7f7a280d9779.slice/crio-47b76969cc3f0249a1395df937076db8eab6898956c77c746fc1d3e5235ff312 WatchSource:0}: Error finding container 47b76969cc3f0249a1395df937076db8eab6898956c77c746fc1d3e5235ff312: Status 404 returned error can't find the container with id 47b76969cc3f0249a1395df937076db8eab6898956c77c746fc1d3e5235ff312 Jun 10 10:45:34.776808 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.776783 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:45:34.777262 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.776812 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:45:34.777262 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.776812 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:45:34.781134 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.781111 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 10 10:45:34.781263 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.781138 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 10 10:45:34.781263 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.781249 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8hbqv\"" Jun 10 10:45:34.781430 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.781386 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 10 10:45:34.781430 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.781401 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgxv7\"" Jun 10 10:45:34.781565 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.781496 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 10 10:45:34.903189 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.903164 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:34.903352 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:34.903304 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:34.903406 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:34.903367 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:45:36.903351183 +0000 UTC m=+38.552283726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:45:34.928439 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.928414 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dqngk" event={"ID":"8e608297-9570-4900-bba2-7f7a280d9779","Type":"ContainerStarted","Data":"47b76969cc3f0249a1395df937076db8eab6898956c77c746fc1d3e5235ff312"} Jun 10 10:45:34.931508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.931462 2545 generic.go:356] "Generic (PLEG): container finished" podID="db8e86ee-d321-4fdc-be9a-495560a2f399" containerID="1c7254214557781722574e4645bcfd245cc944804e576bd412aa2174c32dc399" exitCode=0 Jun 10 10:45:34.931605 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:34.931513 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerDied","Data":"1c7254214557781722574e4645bcfd245cc944804e576bd412aa2174c32dc399"} Jun 10 10:45:35.934939 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:35.934902 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dqngk" event={"ID":"8e608297-9570-4900-bba2-7f7a280d9779","Type":"ContainerStarted","Data":"95a0601fdf037a56756ce518fe00dcf42fdcfb0478bc7986dde5d4f1f276ddd2"} Jun 10 10:45:35.938090 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:35.938064 2545 generic.go:356] "Generic (PLEG): container finished" podID="db8e86ee-d321-4fdc-be9a-495560a2f399" containerID="ba650a837a8a17ca6f313c932e6fff4595494a733e16474a1df149927fb37b02" exitCode=0 Jun 10 10:45:35.938217 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:35.938116 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerDied","Data":"ba650a837a8a17ca6f313c932e6fff4595494a733e16474a1df149927fb37b02"} Jun 10 10:45:35.974885 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:35.974661 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dqngk" podStartSLOduration=1.5031932449999998 podStartE2EDuration="2.974639163s" totalImagesPullingTime="1.471445918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:35.951106563 +0000 UTC m=+37.600039104" watchObservedRunningTime="2026-06-10 10:45:35.974639163 +0000 UTC m=+37.623571704" Jun 10 10:45:36.916251 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:36.916179 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:36.916385 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:36.916316 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:36.916385 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:36.916376 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:45:40.916361038 +0000 UTC m=+42.565293559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:45:36.943296 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:36.943266 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" event={"ID":"db8e86ee-d321-4fdc-be9a-495560a2f399","Type":"ContainerStarted","Data":"d3232f17fcaf1d3352504266740358180a9f5a756db2929c14d169c142aba1a3"} Jun 10 10:45:36.970829 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:36.970784 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cxkhd" podStartSLOduration=9.990102425 podStartE2EDuration="38.970771092s" totalImagesPullingTime="28.980668667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:45:36.970111371 +0000 UTC m=+38.619043911" watchObservedRunningTime="2026-06-10 10:45:36.970771092 +0000 UTC m=+38.619703611" Jun 10 10:45:40.943879 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:40.943848 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:40.944302 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:40.944005 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:40.944302 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:40.944068 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:45:48.944050423 +0000 UTC m=+50.592982961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:45:48.990654 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:48.990619 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:45:48.991275 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:48.990777 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:45:48.991275 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:45:48.990860 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:46:04.990840849 +0000 UTC m=+66.639773369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:45:58.930886 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:45:58.930860 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pfkh8" Jun 10 10:46:04.491062 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.491018 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:46:04.494925 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.494906 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 10 10:46:04.501738 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:46:04.501718 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 10 10:46:04.501810 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:46:04.501773 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:47:08.501758727 +0000 UTC m=+130.150691247 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : secret "metrics-daemon-secret" not found Jun 10 10:46:04.591425 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.591400 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:46:04.594858 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.594844 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 10 10:46:04.606062 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.606046 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff371f6a-32fb-42d5-96f8-d1f6558bf6fa-original-pull-secret\") pod \"global-pull-secret-syncer-47m9g\" (UID: \"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa\") " pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:46:04.692648 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.692623 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:46:04.696295 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.696279 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 10 10:46:04.707306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.707290 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 10 10:46:04.716189 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.716170 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8h9\" (UniqueName: \"kubernetes.io/projected/89fd8558-befa-433f-9575-27edb9ebdf62-kube-api-access-6d8h9\") pod \"network-check-target-gdqfd\" (UID: \"89fd8558-befa-433f-9575-27edb9ebdf62\") " pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:46:04.796900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.796848 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-47m9g" Jun 10 10:46:04.805462 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.805444 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8hbqv\"" Jun 10 10:46:04.812347 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.812329 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:46:04.981572 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.981544 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-47m9g"] Jun 10 10:46:04.984098 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:46:04.984074 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff371f6a_32fb_42d5_96f8_d1f6558bf6fa.slice/crio-b4080434d300ccf41dc893744f851bcefaa8283544c065f6e03b6e5bdb326817 WatchSource:0}: Error finding container b4080434d300ccf41dc893744f851bcefaa8283544c065f6e03b6e5bdb326817: Status 404 returned error can't find the container with id b4080434d300ccf41dc893744f851bcefaa8283544c065f6e03b6e5bdb326817 Jun 10 10:46:04.997055 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:04.997031 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:46:04.997203 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:46:04.997186 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:46:04.997265 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:46:04.997255 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:46:36.997234621 +0000 UTC m=+98.646167140 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:46:05.005661 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:05.005639 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-47m9g" event={"ID":"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa","Type":"ContainerStarted","Data":"b4080434d300ccf41dc893744f851bcefaa8283544c065f6e03b6e5bdb326817"} Jun 10 10:46:05.009498 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:05.009477 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gdqfd"] Jun 10 10:46:05.023932 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:46:05.023908 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fd8558_befa_433f_9575_27edb9ebdf62.slice/crio-cea0e4960bf66b2ebdb91cff1d2b385e035f7bf0b674b9c1cc3066a807858f8f WatchSource:0}: Error finding container cea0e4960bf66b2ebdb91cff1d2b385e035f7bf0b674b9c1cc3066a807858f8f: Status 404 returned error can't find the container with id cea0e4960bf66b2ebdb91cff1d2b385e035f7bf0b674b9c1cc3066a807858f8f Jun 10 10:46:06.010675 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:06.010635 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gdqfd" event={"ID":"89fd8558-befa-433f-9575-27edb9ebdf62","Type":"ContainerStarted","Data":"cea0e4960bf66b2ebdb91cff1d2b385e035f7bf0b674b9c1cc3066a807858f8f"} Jun 10 10:46:10.022458 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:10.022421 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-47m9g" event={"ID":"ff371f6a-32fb-42d5-96f8-d1f6558bf6fa","Type":"ContainerStarted","Data":"2b3835b7b704bf046e4866e29ba9d5bfda50eedeeedd81ec72cce223201a4293"} Jun 10 10:46:10.023814 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:10.023789 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gdqfd" event={"ID":"89fd8558-befa-433f-9575-27edb9ebdf62","Type":"ContainerStarted","Data":"f7eedf7476258d927df67811f1025dfcc4079d653614927d150f1ac0eb595f8a"} Jun 10 10:46:10.023925 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:10.023909 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:46:10.066136 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:10.066098 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-47m9g" podStartSLOduration=66.77541644 podStartE2EDuration="1m11.066086146s" totalImagesPullingTime="4.290669706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:46:10.043524972 +0000 UTC m=+71.692457512" watchObservedRunningTime="2026-06-10 10:46:10.066086146 +0000 UTC m=+71.715018687" Jun 10 10:46:37.096216 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:37.096138 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:46:37.096583 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:46:37.096267 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:46:37.096583 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:46:37.096344 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:47:41.096330577 +0000 UTC m=+162.745263100 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:46:41.027879 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:41.027851 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gdqfd" Jun 10 10:46:41.047279 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:41.047231 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gdqfd" podStartSLOduration=97.801012065 podStartE2EDuration="1m42.047214695s" totalImagesPullingTime="4.24620263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:46:10.067281825 +0000 UTC m=+71.716214366" watchObservedRunningTime="2026-06-10 10:46:41.047214695 +0000 UTC m=+102.696147237" Jun 10 10:46:47.015070 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.015029 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5"] Jun 10 10:46:47.018521 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.018503 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.020662 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.020640 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8"] Jun 10 10:46:47.023130 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.023112 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.023223 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.023212 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jun 10 10:46:47.023284 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.023236 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jun 10 10:46:47.023381 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.023240 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jun 10 10:46:47.023616 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.023593 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jun 10 10:46:47.025994 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.025978 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jun 10 10:46:47.026271 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.026254 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jun 10 10:46:47.027033 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.027018 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jun 10 10:46:47.027101 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.027062 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jun 10 10:46:47.031888 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.031871 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5"] Jun 10 10:46:47.047715 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.047697 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8"] Jun 10 10:46:47.056460 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056441 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/421b7017-6be8-4c42-ac13-956ffe289906-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.056553 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056469 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ljd\" (UniqueName: \"kubernetes.io/projected/a99a27d5-c979-418c-ac00-ea630e1206e5-kube-api-access-58ljd\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.056553 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056488 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a99a27d5-c979-418c-ac00-ea630e1206e5-tmp\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.056553 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056511 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a99a27d5-c979-418c-ac00-ea630e1206e5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.056751 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056556 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvbf8\" (UniqueName: \"kubernetes.io/projected/421b7017-6be8-4c42-ac13-956ffe289906-kube-api-access-lvbf8\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.056751 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056684 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.056751 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056727 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-hub\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.056885 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056776 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-ca\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.056885 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.056800 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.157700 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157664 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/421b7017-6be8-4c42-ac13-956ffe289906-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.157700 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157696 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ljd\" (UniqueName: \"kubernetes.io/projected/a99a27d5-c979-418c-ac00-ea630e1206e5-kube-api-access-58ljd\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.157898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157712 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a99a27d5-c979-418c-ac00-ea630e1206e5-tmp\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.157898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157732 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a99a27d5-c979-418c-ac00-ea630e1206e5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.157898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157760 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvbf8\" (UniqueName: \"kubernetes.io/projected/421b7017-6be8-4c42-ac13-956ffe289906-kube-api-access-lvbf8\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.157898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157814 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.157898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157843 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-hub\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.157898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157871 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-ca\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.157898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.157892 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.158371 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.158339 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a99a27d5-c979-418c-ac00-ea630e1206e5-tmp\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.158528 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.158503 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/421b7017-6be8-4c42-ac13-956ffe289906-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.160340 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.160317 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-ca\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.160340 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.160336 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.160492 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.160351 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a99a27d5-c979-418c-ac00-ea630e1206e5-klusterlet-config\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.160679 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.160663 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.160733 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.160662 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/421b7017-6be8-4c42-ac13-956ffe289906-hub\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.171833 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.171808 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvbf8\" (UniqueName: \"kubernetes.io/projected/421b7017-6be8-4c42-ac13-956ffe289906-kube-api-access-lvbf8\") pod \"cluster-proxy-proxy-agent-7ff8756d89-ztgg8\" (UID: \"421b7017-6be8-4c42-ac13-956ffe289906\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.172890 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.172867 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ljd\" (UniqueName: \"kubernetes.io/projected/a99a27d5-c979-418c-ac00-ea630e1206e5-kube-api-access-58ljd\") pod \"klusterlet-addon-workmgr-5858c4dcf7-z5ht5\" (UID: \"a99a27d5-c979-418c-ac00-ea630e1206e5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.329121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.329027 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:47.345081 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.345059 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:46:47.458949 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.458917 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5"] Jun 10 10:46:47.463997 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:46:47.463956 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99a27d5_c979_418c_ac00_ea630e1206e5.slice/crio-df15f178d0a77882b68fccc11c97a407c7b21c0226756c08b8dd9c06a26e9660 WatchSource:0}: Error finding container df15f178d0a77882b68fccc11c97a407c7b21c0226756c08b8dd9c06a26e9660: Status 404 returned error can't find the container with id df15f178d0a77882b68fccc11c97a407c7b21c0226756c08b8dd9c06a26e9660 Jun 10 10:46:47.479640 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:47.479621 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8"] Jun 10 10:46:47.481668 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:46:47.481649 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod421b7017_6be8_4c42_ac13_956ffe289906.slice/crio-4f4fd649f4c0eb77da2fd7f735629daf1c5d3e5716c5b30924b151577564a854 WatchSource:0}: Error finding container 4f4fd649f4c0eb77da2fd7f735629daf1c5d3e5716c5b30924b151577564a854: Status 404 returned error can't find the container with id 4f4fd649f4c0eb77da2fd7f735629daf1c5d3e5716c5b30924b151577564a854 Jun 10 10:46:48.110988 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:48.110944 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" event={"ID":"a99a27d5-c979-418c-ac00-ea630e1206e5","Type":"ContainerStarted","Data":"df15f178d0a77882b68fccc11c97a407c7b21c0226756c08b8dd9c06a26e9660"} Jun 10 10:46:48.111983 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:48.111949 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" event={"ID":"421b7017-6be8-4c42-ac13-956ffe289906","Type":"ContainerStarted","Data":"4f4fd649f4c0eb77da2fd7f735629daf1c5d3e5716c5b30924b151577564a854"} Jun 10 10:46:53.129692 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.129653 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" event={"ID":"a99a27d5-c979-418c-ac00-ea630e1206e5","Type":"ContainerStarted","Data":"e180cb9f72489e2226a59ee7c70a83064802a4f69e7bd29a968cce2ac9c80de4"} Jun 10 10:46:53.130188 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.129800 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:53.131605 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.131586 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:46:53.131765 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.131747 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" event={"ID":"421b7017-6be8-4c42-ac13-956ffe289906","Type":"ContainerStarted","Data":"6c78c3951cc2910da20aacbab6bec4565483236cceca179fe866beeccb06a44b"} Jun 10 10:46:53.131816 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.131772 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" event={"ID":"421b7017-6be8-4c42-ac13-956ffe289906","Type":"ContainerStarted","Data":"a90c9b8176b152a563adb04d8c9197f495d9e7764bc19c59be0c58604a5e7bd2"} Jun 10 10:46:53.131816 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.131783 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" event={"ID":"421b7017-6be8-4c42-ac13-956ffe289906","Type":"ContainerStarted","Data":"d2325f8bd0723af343e6947232725c7896a67665c5ecbf243d8de82257e8ffd6"} Jun 10 10:46:53.149544 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.149508 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" podStartSLOduration=2.053471358 podStartE2EDuration="7.149495478s" totalImagesPullingTime="5.09602412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:46:53.149120247 +0000 UTC m=+114.798052787" watchObservedRunningTime="2026-06-10 10:46:53.149495478 +0000 UTC m=+114.798428047" Jun 10 10:46:53.188565 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:46:53.188525 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" podStartSLOduration=2.10823921 podStartE2EDuration="7.188512901s" totalImagesPullingTime="5.080273691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:46:53.187491546 +0000 UTC m=+114.836424085" watchObservedRunningTime="2026-06-10 10:46:53.188512901 +0000 UTC m=+114.837445474" Jun 10 10:47:08.512699 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:47:08.512659 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:47:08.513122 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:47:08.512775 2545 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jun 10 10:47:08.513122 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:47:08.512829 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs podName:452be7ad-2d9a-4448-899a-d5e68cc10f2e nodeName:}" failed. No retries permitted until 2026-06-10 10:49:10.512815641 +0000 UTC m=+252.161748161 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs") pod "network-metrics-daemon-6xj9p" (UID: "452be7ad-2d9a-4448-899a-d5e68cc10f2e") : secret "metrics-daemon-secret" not found Jun 10 10:47:36.201097 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:47:36.201050 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tc8c9" podUID="252cfbef-2c8f-4d25-971d-cf008884454d" Jun 10 10:47:36.253409 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:47:36.253380 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tc8c9" Jun 10 10:47:37.788730 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:47:37.788694 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6xj9p" podUID="452be7ad-2d9a-4448-899a-d5e68cc10f2e" Jun 10 10:47:41.140411 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:47:41.140341 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:47:41.140740 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:47:41.140496 2545 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 10 10:47:41.140740 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:47:41.140570 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls podName:252cfbef-2c8f-4d25-971d-cf008884454d nodeName:}" failed. No retries permitted until 2026-06-10 10:49:43.140556396 +0000 UTC m=+284.789488915 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls") pod "dns-default-tc8c9" (UID: "252cfbef-2c8f-4d25-971d-cf008884454d") : secret "dns-default-metrics-tls" not found Jun 10 10:47:50.777007 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:47:50.776958 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:48:21.114797 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:21.114766 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p4l5s_c989fdb9-e253-413c-b23f-8d6c1389d16c/dns-node-resolver/0.log" Jun 10 10:48:21.913926 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:21.913900 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jgp8h_64be67c0-7c44-4334-b8d7-29aa305f5fd5/node-ca/0.log" Jun 10 10:48:22.715276 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:22.715251 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dqngk_8e608297-9570-4900-bba2-7f7a280d9779/serve-healthcheck-canary/0.log" Jun 10 10:48:38.945497 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.945464 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sx4p2"] Jun 10 10:48:38.948464 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.948437 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:38.951764 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.951745 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jun 10 10:48:38.952555 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.952540 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jun 10 10:48:38.952632 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.952553 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4h7qz\"" Jun 10 10:48:38.953206 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.953178 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jun 10 10:48:38.953320 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.953302 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jun 10 10:48:38.965582 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:38.965555 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sx4p2"] Jun 10 10:48:39.048755 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.048717 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba682d21-f1a8-4129-b14a-dd72ceb88452-crio-socket\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.048755 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.048759 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba682d21-f1a8-4129-b14a-dd72ceb88452-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.049034 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.048778 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tnbs\" (UniqueName: \"kubernetes.io/projected/ba682d21-f1a8-4129-b14a-dd72ceb88452-kube-api-access-8tnbs\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.049034 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.048870 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba682d21-f1a8-4129-b14a-dd72ceb88452-data-volume\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.049034 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.048946 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba682d21-f1a8-4129-b14a-dd72ceb88452-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150165 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150130 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba682d21-f1a8-4129-b14a-dd72ceb88452-crio-socket\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150171 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba682d21-f1a8-4129-b14a-dd72ceb88452-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150195 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tnbs\" (UniqueName: \"kubernetes.io/projected/ba682d21-f1a8-4129-b14a-dd72ceb88452-kube-api-access-8tnbs\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150221 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba682d21-f1a8-4129-b14a-dd72ceb88452-data-volume\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150264 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba682d21-f1a8-4129-b14a-dd72ceb88452-crio-socket\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150293 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba682d21-f1a8-4129-b14a-dd72ceb88452-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150643 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150621 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba682d21-f1a8-4129-b14a-dd72ceb88452-data-volume\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.150879 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.150863 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba682d21-f1a8-4129-b14a-dd72ceb88452-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.152475 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.152453 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba682d21-f1a8-4129-b14a-dd72ceb88452-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.181187 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.181154 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tnbs\" (UniqueName: \"kubernetes.io/projected/ba682d21-f1a8-4129-b14a-dd72ceb88452-kube-api-access-8tnbs\") pod \"insights-runtime-extractor-sx4p2\" (UID: \"ba682d21-f1a8-4129-b14a-dd72ceb88452\") " pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.256668 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.256582 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sx4p2" Jun 10 10:48:39.379870 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.379832 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sx4p2"] Jun 10 10:48:39.384009 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:48:39.383981 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba682d21_f1a8_4129_b14a_dd72ceb88452.slice/crio-8abc07301e8affa9a6a80571529f459bad915eeabaefc50bb1087210d0790538 WatchSource:0}: Error finding container 8abc07301e8affa9a6a80571529f459bad915eeabaefc50bb1087210d0790538: Status 404 returned error can't find the container with id 8abc07301e8affa9a6a80571529f459bad915eeabaefc50bb1087210d0790538 Jun 10 10:48:39.420830 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:39.420799 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx4p2" event={"ID":"ba682d21-f1a8-4129-b14a-dd72ceb88452","Type":"ContainerStarted","Data":"8abc07301e8affa9a6a80571529f459bad915eeabaefc50bb1087210d0790538"} Jun 10 10:48:40.426041 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.426003 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx4p2" event={"ID":"ba682d21-f1a8-4129-b14a-dd72ceb88452","Type":"ContainerStarted","Data":"2d7e4d3f1034da4b833b18a43242449b2e89fba76a4f843a85469622a669afe8"} Jun 10 10:48:40.426463 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.426065 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx4p2" event={"ID":"ba682d21-f1a8-4129-b14a-dd72ceb88452","Type":"ContainerStarted","Data":"a46b4c341b11536187271483a67a8467e5fcc5bebd64f06615dc81014e3c0647"} Jun 10 10:48:40.694674 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.694595 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6657875f97-g4jrs"] Jun 10 10:48:40.697730 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.697707 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6657875f97-g4jrs" Jun 10 10:48:40.700781 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.700759 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jun 10 10:48:40.700915 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.700759 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8dzqb\"" Jun 10 10:48:40.700915 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.700762 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jun 10 10:48:40.708508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.708485 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6657875f97-g4jrs"] Jun 10 10:48:40.765273 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.765231 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbm97\" (UniqueName: \"kubernetes.io/projected/fdf31123-573d-4704-b464-1c1cabd7b5e9-kube-api-access-gbm97\") pod \"downloads-6657875f97-g4jrs\" (UID: \"fdf31123-573d-4704-b464-1c1cabd7b5e9\") " pod="openshift-console/downloads-6657875f97-g4jrs" Jun 10 10:48:40.866714 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.866679 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbm97\" (UniqueName: \"kubernetes.io/projected/fdf31123-573d-4704-b464-1c1cabd7b5e9-kube-api-access-gbm97\") pod \"downloads-6657875f97-g4jrs\" (UID: \"fdf31123-573d-4704-b464-1c1cabd7b5e9\") " pod="openshift-console/downloads-6657875f97-g4jrs" Jun 10 10:48:40.876517 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:40.876487 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbm97\" (UniqueName: \"kubernetes.io/projected/fdf31123-573d-4704-b464-1c1cabd7b5e9-kube-api-access-gbm97\") pod \"downloads-6657875f97-g4jrs\" (UID: \"fdf31123-573d-4704-b464-1c1cabd7b5e9\") " pod="openshift-console/downloads-6657875f97-g4jrs" Jun 10 10:48:41.008402 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:41.008317 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6657875f97-g4jrs" Jun 10 10:48:41.422881 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:41.422849 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6657875f97-g4jrs"] Jun 10 10:48:41.425921 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:48:41.425876 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf31123_573d_4704_b464_1c1cabd7b5e9.slice/crio-669feae93d68f0f5f4ffa969243f60fac5dcdae0cb5ca16f8a2894fc4ded05eb WatchSource:0}: Error finding container 669feae93d68f0f5f4ffa969243f60fac5dcdae0cb5ca16f8a2894fc4ded05eb: Status 404 returned error can't find the container with id 669feae93d68f0f5f4ffa969243f60fac5dcdae0cb5ca16f8a2894fc4ded05eb Jun 10 10:48:41.430477 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:41.430446 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6657875f97-g4jrs" event={"ID":"fdf31123-573d-4704-b464-1c1cabd7b5e9","Type":"ContainerStarted","Data":"669feae93d68f0f5f4ffa969243f60fac5dcdae0cb5ca16f8a2894fc4ded05eb"} Jun 10 10:48:41.432388 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:41.432358 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx4p2" event={"ID":"ba682d21-f1a8-4129-b14a-dd72ceb88452","Type":"ContainerStarted","Data":"a2ddfc3cdaec97e25324b6030a79a5e62e9e03278e991a66021e82c38121f6ba"} Jun 10 10:48:41.452695 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:41.452641 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sx4p2" podStartSLOduration=1.612323967 podStartE2EDuration="3.452626141s" totalImagesPullingTime="1.840302174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 10:48:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:41.451786034 +0000 UTC m=+223.100718575" watchObservedRunningTime="2026-06-10 10:48:41.452626141 +0000 UTC m=+223.101558682" Jun 10 10:48:51.235885 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.235846 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d5d87f77f-g5wf4"] Jun 10 10:48:51.239960 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.239931 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.244217 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.244190 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jun 10 10:48:51.244217 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.244205 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jun 10 10:48:51.244402 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.244240 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jun 10 10:48:51.244402 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.244207 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jun 10 10:48:51.244402 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.244190 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qzwfh\"" Jun 10 10:48:51.244548 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.244444 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jun 10 10:48:51.249543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.249491 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d5d87f77f-g5wf4"] Jun 10 10:48:51.354073 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.354030 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgcm\" (UniqueName: \"kubernetes.io/projected/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-kube-api-access-hcgcm\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.354246 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.354156 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-service-ca\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.354304 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.354238 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-oauth-config\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.354350 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.354297 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-config\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.354350 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.354328 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-serving-cert\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.354440 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.354396 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-oauth-serving-cert\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.454783 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.454746 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-config\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.454783 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.454781 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-serving-cert\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.455042 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.454828 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-oauth-serving-cert\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.455042 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.454850 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgcm\" (UniqueName: \"kubernetes.io/projected/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-kube-api-access-hcgcm\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.455148 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.455059 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-service-ca\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.455148 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.455103 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-oauth-config\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.455779 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.455754 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-config\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.455779 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.455772 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-oauth-serving-cert\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.455933 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.455791 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-service-ca\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.457714 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.457693 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-serving-cert\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.457832 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.457751 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-oauth-config\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.464364 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.464339 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgcm\" (UniqueName: \"kubernetes.io/projected/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-kube-api-access-hcgcm\") pod \"console-d5d87f77f-g5wf4\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:51.550664 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:51.550579 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:48:53.130476 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:53.130419 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" podUID="a99a27d5-c979-418c-ac00-ea630e1206e5" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused" Jun 10 10:48:54.775604 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.775571 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mnwsx"] Jun 10 10:48:54.778950 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.778924 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.782675 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.782551 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jun 10 10:48:54.782675 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.782637 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jun 10 10:48:54.782891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.782775 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jun 10 10:48:54.784027 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.784002 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jun 10 10:48:54.784027 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.784013 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jun 10 10:48:54.784200 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.784168 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-859ww\"" Jun 10 10:48:54.789263 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.789244 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jun 10 10:48:54.880717 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880675 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-tls\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.880717 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880717 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-textfile\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.880990 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880744 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-sys\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.880990 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880768 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.880990 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880787 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-wtmp\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.880990 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880803 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-metrics-client-ca\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.880990 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880919 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.880990 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.880954 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxxf\" (UniqueName: \"kubernetes.io/projected/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-kube-api-access-6vxxf\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.881292 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.881033 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-root\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982058 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982004 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982058 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982054 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxxf\" (UniqueName: \"kubernetes.io/projected/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-kube-api-access-6vxxf\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982104 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-root\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982128 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-tls\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982149 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-textfile\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982171 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-sys\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982196 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982224 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-wtmp\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982227 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-root\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982313 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982249 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-metrics-client-ca\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982705 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982515 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-textfile\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982705 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982685 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-accelerators-collector-config\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982815 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982745 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-sys\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.982815 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982744 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-wtmp\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.983008 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.982984 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-metrics-client-ca\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.986309 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.986247 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.986747 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.986721 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-node-exporter-tls\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:54.994104 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:54.994073 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxxf\" (UniqueName: \"kubernetes.io/projected/c80e2331-c8d3-4bf5-8c4d-6047c3652f93-kube-api-access-6vxxf\") pod \"node-exporter-mnwsx\" (UID: \"c80e2331-c8d3-4bf5-8c4d-6047c3652f93\") " pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:55.090398 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.090304 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mnwsx" Jun 10 10:48:55.795706 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.795662 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:48:55.799338 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.799316 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.802497 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.802472 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 10 10:48:55.803120 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803049 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 10 10:48:55.803120 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803098 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 10 10:48:55.803120 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803117 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 10 10:48:55.803326 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803194 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 10 10:48:55.803326 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803205 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6bl24\"" Jun 10 10:48:55.803597 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803580 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 10 10:48:55.803679 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803595 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 10 10:48:55.803679 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803607 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 10 10:48:55.803679 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.803596 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 10 10:48:55.815602 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.815574 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:48:55.890226 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890184 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-web-config\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890384 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890236 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890384 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890274 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890384 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890302 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890384 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890332 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890384 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890358 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890384 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890380 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890662 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890419 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890662 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890450 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hts\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-kube-api-access-t9hts\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890662 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890482 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890662 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890597 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890662 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890647 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-config-out\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.890857 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.890713 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.992806 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.992767 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.992823 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-web-config\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.992853 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.992884 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.992912 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.992942 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:48:55.992958 2545 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.992982 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993006 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993036 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993032 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993477 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:48:55.993050 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls podName:bd396cfd-5d99-425b-ad86-36052cfd5035 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:56.493031292 +0000 UTC m=+238.141963831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035") : secret "alertmanager-main-tls" not found Jun 10 10:48:55.993477 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993088 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hts\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-kube-api-access-t9hts\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993477 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993132 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993477 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993185 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993477 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993222 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-config-out\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993767 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993742 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.993999 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.993935 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.994604 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.994564 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.996701 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.996672 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.996884 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.996836 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-config-out\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.997080 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.997035 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.997269 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.997247 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.997954 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.997915 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-web-config\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.998059 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.998009 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.998390 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.998361 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:55.998478 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:55.998416 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:56.011517 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:56.011490 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hts\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-kube-api-access-t9hts\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:56.498602 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:56.498558 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:56.498808 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:48:56.498747 2545 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jun 10 10:48:56.498866 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:48:56.498827 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls podName:bd396cfd-5d99-425b-ad86-36052cfd5035 nodeName:}" failed. No retries permitted until 2026-06-10 10:48:57.498804987 +0000 UTC m=+239.147737510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035") : secret "alertmanager-main-tls" not found Jun 10 10:48:56.990789 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:48:56.990756 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80e2331_c8d3_4bf5_8c4d_6047c3652f93.slice/crio-a60df8b7b8f909292403082e6a420a6f0deaff51c9a3580b287a16c0f6c0afa9 WatchSource:0}: Error finding container a60df8b7b8f909292403082e6a420a6f0deaff51c9a3580b287a16c0f6c0afa9: Status 404 returned error can't find the container with id a60df8b7b8f909292403082e6a420a6f0deaff51c9a3580b287a16c0f6c0afa9 Jun 10 10:48:57.116051 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.115940 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d5d87f77f-g5wf4"] Jun 10 10:48:57.118619 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:48:57.118591 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02b0a61_0a66_4c9b_9f76_1aa3fbcee8a9.slice/crio-c5fed4823ddf196b60243a36bf1418488031decea037970fd1680dcc75353084 WatchSource:0}: Error finding container c5fed4823ddf196b60243a36bf1418488031decea037970fd1680dcc75353084: Status 404 returned error can't find the container with id c5fed4823ddf196b60243a36bf1418488031decea037970fd1680dcc75353084 Jun 10 10:48:57.330265 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.330166 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" podUID="a99a27d5-c979-418c-ac00-ea630e1206e5" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/healthz\": dial tcp 10.134.0.8:8000: connect: connection refused" Jun 10 10:48:57.494767 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.494151 2545 generic.go:356] "Generic (PLEG): container finished" podID="a99a27d5-c979-418c-ac00-ea630e1206e5" containerID="e180cb9f72489e2226a59ee7c70a83064802a4f69e7bd29a968cce2ac9c80de4" exitCode=1 Jun 10 10:48:57.494767 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.494236 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" event={"ID":"a99a27d5-c979-418c-ac00-ea630e1206e5","Type":"ContainerDied","Data":"e180cb9f72489e2226a59ee7c70a83064802a4f69e7bd29a968cce2ac9c80de4"} Jun 10 10:48:57.494767 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.494638 2545 scope.go:117] "RemoveContainer" containerID="e180cb9f72489e2226a59ee7c70a83064802a4f69e7bd29a968cce2ac9c80de4" Jun 10 10:48:57.498731 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.498702 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6657875f97-g4jrs" event={"ID":"fdf31123-573d-4704-b464-1c1cabd7b5e9","Type":"ContainerStarted","Data":"49d3f6bcb37e952961fd800cf47c046187c4f7feb08eb9a882a0c47db4824685"} Jun 10 10:48:57.501738 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.501649 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6657875f97-g4jrs" Jun 10 10:48:57.503527 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.503501 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnwsx" event={"ID":"c80e2331-c8d3-4bf5-8c4d-6047c3652f93","Type":"ContainerStarted","Data":"a60df8b7b8f909292403082e6a420a6f0deaff51c9a3580b287a16c0f6c0afa9"} Jun 10 10:48:57.505706 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.505673 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d87f77f-g5wf4" event={"ID":"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9","Type":"ContainerStarted","Data":"c5fed4823ddf196b60243a36bf1418488031decea037970fd1680dcc75353084"} Jun 10 10:48:57.507868 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.507840 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:57.513189 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.512524 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:57.516760 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.516707 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6657875f97-g4jrs" Jun 10 10:48:57.535483 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.535418 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6657875f97-g4jrs" podStartSLOduration=1.8877650529999999 podStartE2EDuration="17.535395695s" totalImagesPullingTime="15.647630642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:57.533858986 +0000 UTC m=+239.182791536" watchObservedRunningTime="2026-06-10 10:48:57.535395695 +0000 UTC m=+239.184328237" Jun 10 10:48:57.610532 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.610270 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:48:57.834980 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:57.834848 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:48:58.019259 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:48:58.019177 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd396cfd_5d99_425b_ad86_36052cfd5035.slice/crio-d2b48154a5ef60e884a3692eb0e41163d32ce11606c55404d6257fdeafef3221 WatchSource:0}: Error finding container d2b48154a5ef60e884a3692eb0e41163d32ce11606c55404d6257fdeafef3221: Status 404 returned error can't find the container with id d2b48154a5ef60e884a3692eb0e41163d32ce11606c55404d6257fdeafef3221 Jun 10 10:48:58.517527 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:58.517487 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" event={"ID":"a99a27d5-c979-418c-ac00-ea630e1206e5","Type":"ContainerStarted","Data":"48a9f8ab9a9cd62efe9a54d2fbd37e07e2b192ceb9b7a629b5506e7912415c06"} Jun 10 10:48:58.519000 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:58.518518 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:48:58.520003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:58.519940 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5858c4dcf7-z5ht5" Jun 10 10:48:58.523450 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:58.522711 2545 generic.go:356] "Generic (PLEG): container finished" podID="c80e2331-c8d3-4bf5-8c4d-6047c3652f93" containerID="1bbb063a170ae1ebb43433b0729e95a6f689183de76aba6ffcd7db0a99c31c1b" exitCode=0 Jun 10 10:48:58.523450 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:58.522810 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnwsx" event={"ID":"c80e2331-c8d3-4bf5-8c4d-6047c3652f93","Type":"ContainerDied","Data":"1bbb063a170ae1ebb43433b0729e95a6f689183de76aba6ffcd7db0a99c31c1b"} Jun 10 10:48:58.527648 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:58.527603 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerStarted","Data":"d2b48154a5ef60e884a3692eb0e41163d32ce11606c55404d6257fdeafef3221"} Jun 10 10:48:59.443635 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.443388 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-76b68cc874-m9c65"] Jun 10 10:48:59.469858 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.469800 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-76b68cc874-m9c65"] Jun 10 10:48:59.470032 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.469999 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:48:59.470111 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.470005 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d5d87f77f-g5wf4"] Jun 10 10:48:59.473443 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.473220 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-spdwx\"" Jun 10 10:48:59.473581 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.473565 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jun 10 10:48:59.534935 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.534790 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnwsx" event={"ID":"c80e2331-c8d3-4bf5-8c4d-6047c3652f93","Type":"ContainerStarted","Data":"393866d52be627702f4e6bb9a57902375239217793c9054c7e6479b87a5c1743"} Jun 10 10:48:59.534935 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.534859 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mnwsx" event={"ID":"c80e2331-c8d3-4bf5-8c4d-6047c3652f93","Type":"ContainerStarted","Data":"8497ce66f6bf3d2e2e637f6a15b23b3dc4330a3bcb40dc9e0b76966b3276f886"} Jun 10 10:48:59.555861 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.555795 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mnwsx" podStartSLOduration=4.389631581 podStartE2EDuration="5.555774294s" totalImagesPullingTime="1.076664841s" totalInitContainerRuntime="89.477872ms" isStatefulPod=true podCreationTimestamp="2026-06-10 10:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:48:59.554983105 +0000 UTC m=+241.203915643" watchObservedRunningTime="2026-06-10 10:48:59.555774294 +0000 UTC m=+241.204706836" Jun 10 10:48:59.629182 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.629137 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-monitoring-plugin-cert\") pod \"monitoring-plugin-76b68cc874-m9c65\" (UID: \"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:48:59.629182 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.629188 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdphw\" (UniqueName: \"kubernetes.io/projected/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-kube-api-access-jdphw\") pod \"monitoring-plugin-76b68cc874-m9c65\" (UID: \"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:48:59.730610 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.730405 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-monitoring-plugin-cert\") pod \"monitoring-plugin-76b68cc874-m9c65\" (UID: \"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:48:59.730610 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:48:59.730562 2545 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jun 10 10:48:59.730831 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:48:59.730644 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-monitoring-plugin-cert podName:fabe843c-fbeb-4b61-ba3c-424fafe0a1b6 nodeName:}" failed. No retries permitted until 2026-06-10 10:49:00.230622179 +0000 UTC m=+241.879554698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-monitoring-plugin-cert") pod "monitoring-plugin-76b68cc874-m9c65" (UID: "fabe843c-fbeb-4b61-ba3c-424fafe0a1b6") : secret "monitoring-plugin-cert" not found Jun 10 10:48:59.731127 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.730907 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdphw\" (UniqueName: \"kubernetes.io/projected/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-kube-api-access-jdphw\") pod \"monitoring-plugin-76b68cc874-m9c65\" (UID: \"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:48:59.742003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.741951 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdphw\" (UniqueName: \"kubernetes.io/projected/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-kube-api-access-jdphw\") pod \"monitoring-plugin-76b68cc874-m9c65\" (UID: \"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:48:59.915557 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.914181 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-584f4f48dc-gbxgf"] Jun 10 10:48:59.939673 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.939618 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:48:59.944462 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.944432 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jun 10 10:48:59.945067 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.945045 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jun 10 10:48:59.945543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.945371 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nd4ph\"" Jun 10 10:48:59.945543 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.945415 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jun 10 10:48:59.945718 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.945703 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jun 10 10:48:59.946369 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.946311 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jun 10 10:48:59.953825 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.953803 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jun 10 10:48:59.963306 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:48:59.963147 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-584f4f48dc-gbxgf"] Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135101 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-telemeter-client-tls\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135171 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-federate-client-tls\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135224 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-telemeter-trusted-ca-bundle\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135295 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-metrics-client-ca\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135320 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrps6\" (UniqueName: \"kubernetes.io/projected/904a7605-4da6-4447-882b-a20cea57ab98-kube-api-access-mrps6\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135347 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-secret-telemeter-client\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135376 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-serving-certs-ca-bundle\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.135508 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.135404 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.236810 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.236771 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-telemeter-client-tls\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.237046 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.236854 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-federate-client-tls\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.237046 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.236904 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-telemeter-trusted-ca-bundle\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.237046 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.236942 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-monitoring-plugin-cert\") pod \"monitoring-plugin-76b68cc874-m9c65\" (UID: \"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:49:00.237046 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.236989 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-metrics-client-ca\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.237046 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.237024 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrps6\" (UniqueName: \"kubernetes.io/projected/904a7605-4da6-4447-882b-a20cea57ab98-kube-api-access-mrps6\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.237338 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.237055 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-secret-telemeter-client\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.237338 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.237087 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-serving-certs-ca-bundle\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.237338 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.237113 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.238099 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.238008 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-telemeter-trusted-ca-bundle\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.239753 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.239728 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-metrics-client-ca\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.240201 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.240153 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.240333 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.240308 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/904a7605-4da6-4447-882b-a20cea57ab98-serving-certs-ca-bundle\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.241582 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.241556 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fabe843c-fbeb-4b61-ba3c-424fafe0a1b6-monitoring-plugin-cert\") pod \"monitoring-plugin-76b68cc874-m9c65\" (UID: \"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6\") " pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:49:00.242129 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.242082 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-federate-client-tls\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.242805 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.242483 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-telemeter-client-tls\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.242805 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.242764 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/904a7605-4da6-4447-882b-a20cea57ab98-secret-telemeter-client\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.252997 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.252931 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrps6\" (UniqueName: \"kubernetes.io/projected/904a7605-4da6-4447-882b-a20cea57ab98-kube-api-access-mrps6\") pod \"telemeter-client-584f4f48dc-gbxgf\" (UID: \"904a7605-4da6-4447-882b-a20cea57ab98\") " pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.253288 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.253264 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" Jun 10 10:49:00.382322 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:00.382285 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:49:01.287737 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.287600 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-76b68cc874-m9c65"] Jun 10 10:49:01.293423 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:49:01.293381 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabe843c_fbeb_4b61_ba3c_424fafe0a1b6.slice/crio-76fedb3993c3085f10cc850793f169c8feabef045d9349b75cfbd2e7e50ecdb0 WatchSource:0}: Error finding container 76fedb3993c3085f10cc850793f169c8feabef045d9349b75cfbd2e7e50ecdb0: Status 404 returned error can't find the container with id 76fedb3993c3085f10cc850793f169c8feabef045d9349b75cfbd2e7e50ecdb0 Jun 10 10:49:01.310411 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.310378 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-584f4f48dc-gbxgf"] Jun 10 10:49:01.311736 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:49:01.311708 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904a7605_4da6_4447_882b_a20cea57ab98.slice/crio-d16c985534f66f73def9be5beca31a71d2d7550e9703deef62de87a960be9868 WatchSource:0}: Error finding container d16c985534f66f73def9be5beca31a71d2d7550e9703deef62de87a960be9868: Status 404 returned error can't find the container with id d16c985534f66f73def9be5beca31a71d2d7550e9703deef62de87a960be9868 Jun 10 10:49:01.543987 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.543923 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d87f77f-g5wf4" event={"ID":"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9","Type":"ContainerStarted","Data":"7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb"} Jun 10 10:49:01.545536 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.545504 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" event={"ID":"904a7605-4da6-4447-882b-a20cea57ab98","Type":"ContainerStarted","Data":"d16c985534f66f73def9be5beca31a71d2d7550e9703deef62de87a960be9868"} Jun 10 10:49:01.546895 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.546859 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" event={"ID":"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6","Type":"ContainerStarted","Data":"76fedb3993c3085f10cc850793f169c8feabef045d9349b75cfbd2e7e50ecdb0"} Jun 10 10:49:01.548555 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.548529 2545 generic.go:356] "Generic (PLEG): container finished" podID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerID="f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e" exitCode=0 Jun 10 10:49:01.548685 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.548616 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e"} Jun 10 10:49:01.551045 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.551022 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:49:01.564766 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:01.564699 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d5d87f77f-g5wf4" podStartSLOduration=6.553291565 podStartE2EDuration="10.5646758s" totalImagesPullingTime="4.011384235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:01.563794809 +0000 UTC m=+243.212727351" watchObservedRunningTime="2026-06-10 10:49:01.5646758 +0000 UTC m=+243.213608342" Jun 10 10:49:05.570311 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.570276 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerStarted","Data":"de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091"} Jun 10 10:49:05.570770 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.570395 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerStarted","Data":"b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6"} Jun 10 10:49:05.570770 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.570431 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerStarted","Data":"5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e"} Jun 10 10:49:05.570770 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.570465 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerStarted","Data":"be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4"} Jun 10 10:49:05.573294 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.572832 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" event={"ID":"904a7605-4da6-4447-882b-a20cea57ab98","Type":"ContainerStarted","Data":"96f3e3f697a1aac091904e4b01b87b5595f650f1a8ecff25914698ebbdf426ef"} Jun 10 10:49:05.573294 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.572888 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" event={"ID":"904a7605-4da6-4447-882b-a20cea57ab98","Type":"ContainerStarted","Data":"063333405dffff83f176c35ad888dbce6da00e6caa96d4dc708fac4d051e219d"} Jun 10 10:49:05.573294 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.572910 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" event={"ID":"904a7605-4da6-4447-882b-a20cea57ab98","Type":"ContainerStarted","Data":"f7cf79681fcdda50a9b5a48e10541c8c578e1ec1afd378cbf0a9ff3d5ebfd8ff"} Jun 10 10:49:05.574859 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.574807 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" event={"ID":"fabe843c-fbeb-4b61-ba3c-424fafe0a1b6","Type":"ContainerStarted","Data":"2d45307657e8791a17a6f10fd7498f6d05f9b263370e9a807eab35ed59da7714"} Jun 10 10:49:05.575045 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.575020 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:49:05.581028 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.580955 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" Jun 10 10:49:05.598560 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.598106 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-584f4f48dc-gbxgf" podStartSLOduration=3.063706498 podStartE2EDuration="6.598090263s" totalImagesPullingTime="3.534383765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:05.595941031 +0000 UTC m=+247.244873572" watchObservedRunningTime="2026-06-10 10:49:05.598090263 +0000 UTC m=+247.247022803" Jun 10 10:49:05.614981 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:05.614917 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-76b68cc874-m9c65" podStartSLOduration=3.068955508 podStartE2EDuration="6.614904087s" totalImagesPullingTime="3.545948579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:05.614205985 +0000 UTC m=+247.263138526" watchObservedRunningTime="2026-06-10 10:49:05.614904087 +0000 UTC m=+247.263836627" Jun 10 10:49:06.580752 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:06.580717 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerStarted","Data":"d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f"} Jun 10 10:49:07.587774 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:07.587733 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerStarted","Data":"baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e"} Jun 10 10:49:07.619379 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:07.619322 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.029296837 podStartE2EDuration="12.619303686s" totalImagesPullingTime="7.570456103s" totalInitContainerRuntime="19.550746ms" isStatefulPod=false podCreationTimestamp="2026-06-10 10:48:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:07.616894796 +0000 UTC m=+249.265827348" watchObservedRunningTime="2026-06-10 10:49:07.619303686 +0000 UTC m=+249.268236225" Jun 10 10:49:10.529912 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:10.529811 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:49:10.532169 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:10.532148 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452be7ad-2d9a-4448-899a-d5e68cc10f2e-metrics-certs\") pod \"network-metrics-daemon-6xj9p\" (UID: \"452be7ad-2d9a-4448-899a-d5e68cc10f2e\") " pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:49:10.581084 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:10.581055 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgxv7\"" Jun 10 10:49:10.588624 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:10.588596 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xj9p" Jun 10 10:49:10.726498 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:10.726452 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xj9p"] Jun 10 10:49:10.729158 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:49:10.729121 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452be7ad_2d9a_4448_899a_d5e68cc10f2e.slice/crio-a1dba2aff776fd0928e86b70dbd8c650027e3900aa1aceba7fa57781ca15c007 WatchSource:0}: Error finding container a1dba2aff776fd0928e86b70dbd8c650027e3900aa1aceba7fa57781ca15c007: Status 404 returned error can't find the container with id a1dba2aff776fd0928e86b70dbd8c650027e3900aa1aceba7fa57781ca15c007 Jun 10 10:49:11.601687 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:11.601649 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xj9p" event={"ID":"452be7ad-2d9a-4448-899a-d5e68cc10f2e","Type":"ContainerStarted","Data":"a1dba2aff776fd0928e86b70dbd8c650027e3900aa1aceba7fa57781ca15c007"} Jun 10 10:49:12.607148 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:12.607113 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xj9p" event={"ID":"452be7ad-2d9a-4448-899a-d5e68cc10f2e","Type":"ContainerStarted","Data":"a8dbb9a83425baddaeac315005f56db383cbe6ae9b625a1fb83594e16e664a0c"} Jun 10 10:49:12.607591 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:12.607165 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xj9p" event={"ID":"452be7ad-2d9a-4448-899a-d5e68cc10f2e","Type":"ContainerStarted","Data":"f4d02cac1a799bfd88beb11cfab2d57d7e9c57b7628b5ecbb16be7121646bbac"} Jun 10 10:49:12.623946 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:12.623895 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6xj9p" podStartSLOduration=253.324354375 podStartE2EDuration="4m14.623878149s" totalImagesPullingTime="1.299523774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:12.623028725 +0000 UTC m=+254.271961265" watchObservedRunningTime="2026-06-10 10:49:12.623878149 +0000 UTC m=+254.272810720" Jun 10 10:49:17.346653 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:17.346615 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" podUID="421b7017-6be8-4c42-ac13-956ffe289906" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jun 10 10:49:26.567227 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.567181 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-d5d87f77f-g5wf4" podUID="a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" containerName="console" containerID="cri-o://7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb" gracePeriod=15 Jun 10 10:49:26.836938 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.836863 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:49:26.974710 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.974666 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcgcm\" (UniqueName: \"kubernetes.io/projected/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-kube-api-access-hcgcm\") pod \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " Jun 10 10:49:26.974902 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.974723 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-service-ca\") pod \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " Jun 10 10:49:26.974902 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.974753 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-oauth-serving-cert\") pod \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " Jun 10 10:49:26.974902 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.974780 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-oauth-config\") pod \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " Jun 10 10:49:26.974902 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.974803 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-serving-cert\") pod \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " Jun 10 10:49:26.974902 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.974828 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-config\") pod \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\" (UID: \"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9\") " Jun 10 10:49:26.975266 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.975233 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-service-ca" (OuterVolumeSpecName: "service-ca") pod "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" (UID: "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:26.975341 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.975240 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" (UID: "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:26.975341 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.975249 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-config" (OuterVolumeSpecName: "console-config") pod "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" (UID: "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:49:26.977125 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.977095 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" (UID: "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:26.977229 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.977117 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" (UID: "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:49:26.977229 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:26.977138 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-kube-api-access-hcgcm" (OuterVolumeSpecName: "kube-api-access-hcgcm") pod "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" (UID: "a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9"). InnerVolumeSpecName "kube-api-access-hcgcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:49:27.076328 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.076293 2545 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-oauth-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:49:27.076328 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.076321 2545 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-oauth-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:49:27.076328 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.076333 2545 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:49:27.076560 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.076342 2545 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-console-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:49:27.076560 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.076351 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgcm\" (UniqueName: \"kubernetes.io/projected/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-kube-api-access-hcgcm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:49:27.076560 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.076359 2545 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:49:27.346776 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.346737 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" podUID="421b7017-6be8-4c42-ac13-956ffe289906" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jun 10 10:49:27.663290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.663252 2545 generic.go:356] "Generic (PLEG): container finished" podID="a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" containerID="7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb" exitCode=0 Jun 10 10:49:27.663290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.663295 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5d87f77f-g5wf4" Jun 10 10:49:27.663795 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.663324 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d87f77f-g5wf4" event={"ID":"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9","Type":"ContainerDied","Data":"7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb"} Jun 10 10:49:27.663795 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.663373 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5d87f77f-g5wf4" event={"ID":"a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9","Type":"ContainerDied","Data":"c5fed4823ddf196b60243a36bf1418488031decea037970fd1680dcc75353084"} Jun 10 10:49:27.663795 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.663403 2545 scope.go:117] "RemoveContainer" containerID="7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb" Jun 10 10:49:27.679802 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.679776 2545 scope.go:117] "RemoveContainer" containerID="7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb" Jun 10 10:49:27.680199 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:49:27.680173 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb\": container with ID starting with 7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb not found: ID does not exist" containerID="7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb" Jun 10 10:49:27.680290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.680205 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb"} err="failed to get container status \"7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb\": rpc error: code = NotFound desc = could not find container \"7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb\": container with ID starting with 7da8f6527340e532e1e17e6498eb51c164f422ad51029287ae719800832517cb not found: ID does not exist" Jun 10 10:49:27.688632 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.688607 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d5d87f77f-g5wf4"] Jun 10 10:49:27.692067 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:27.692044 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d5d87f77f-g5wf4"] Jun 10 10:49:28.780634 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:28.780601 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" path="/var/lib/kubelet/pods/a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9/volumes" Jun 10 10:49:37.346129 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:37.346087 2545 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" podUID="421b7017-6be8-4c42-ac13-956ffe289906" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jun 10 10:49:37.346570 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:37.346166 2545 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" Jun 10 10:49:37.346637 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:37.346603 2545 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6c78c3951cc2910da20aacbab6bec4565483236cceca179fe866beeccb06a44b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jun 10 10:49:37.346683 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:37.346637 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" podUID="421b7017-6be8-4c42-ac13-956ffe289906" containerName="service-proxy" containerID="cri-o://6c78c3951cc2910da20aacbab6bec4565483236cceca179fe866beeccb06a44b" gracePeriod=30 Jun 10 10:49:37.700277 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:37.700241 2545 generic.go:356] "Generic (PLEG): container finished" podID="421b7017-6be8-4c42-ac13-956ffe289906" containerID="6c78c3951cc2910da20aacbab6bec4565483236cceca179fe866beeccb06a44b" exitCode=2 Jun 10 10:49:37.700441 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:37.700318 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" event={"ID":"421b7017-6be8-4c42-ac13-956ffe289906","Type":"ContainerDied","Data":"6c78c3951cc2910da20aacbab6bec4565483236cceca179fe866beeccb06a44b"} Jun 10 10:49:37.700441 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:37.700366 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ff8756d89-ztgg8" event={"ID":"421b7017-6be8-4c42-ac13-956ffe289906","Type":"ContainerStarted","Data":"1ea51eb9e1dad217a7bd6408b27d34b87de8e00bf91e939a32d84f6cc8dc1b83"} Jun 10 10:49:39.254638 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:49:39.254578 2545 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tc8c9" podUID="252cfbef-2c8f-4d25-971d-cf008884454d" Jun 10 10:49:39.709195 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:39.709167 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tc8c9" Jun 10 10:49:43.208492 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:43.208456 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:49:43.210718 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:43.210698 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/252cfbef-2c8f-4d25-971d-cf008884454d-metrics-tls\") pod \"dns-default-tc8c9\" (UID: \"252cfbef-2c8f-4d25-971d-cf008884454d\") " pod="openshift-dns/dns-default-tc8c9" Jun 10 10:49:43.313407 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:43.313369 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-977dh\"" Jun 10 10:49:43.320762 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:43.320740 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tc8c9" Jun 10 10:49:43.463288 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:43.463183 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tc8c9"] Jun 10 10:49:43.465403 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:49:43.465368 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252cfbef_2c8f_4d25_971d_cf008884454d.slice/crio-371719752b89e65fd557daf515b618ae7b2d8963588eb8ba0f8c1904e242bb78 WatchSource:0}: Error finding container 371719752b89e65fd557daf515b618ae7b2d8963588eb8ba0f8c1904e242bb78: Status 404 returned error can't find the container with id 371719752b89e65fd557daf515b618ae7b2d8963588eb8ba0f8c1904e242bb78 Jun 10 10:49:43.723957 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:43.723872 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tc8c9" event={"ID":"252cfbef-2c8f-4d25-971d-cf008884454d","Type":"ContainerStarted","Data":"371719752b89e65fd557daf515b618ae7b2d8963588eb8ba0f8c1904e242bb78"} Jun 10 10:49:45.734809 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:45.734776 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tc8c9" event={"ID":"252cfbef-2c8f-4d25-971d-cf008884454d","Type":"ContainerStarted","Data":"34fc82a7c031d23945d5ed0074c8c60433927003aa076dd60888b96ae72f0456"} Jun 10 10:49:45.735232 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:45.734832 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tc8c9" event={"ID":"252cfbef-2c8f-4d25-971d-cf008884454d","Type":"ContainerStarted","Data":"b3905f58e798a4bf03bab6e8be517f2af90b92714659a79f79c7276e80eed0fe"} Jun 10 10:49:45.735232 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:45.735025 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tc8c9" Jun 10 10:49:45.754480 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:45.754433 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tc8c9" podStartSLOduration=251.073288446 podStartE2EDuration="4m12.754414428s" totalImagesPullingTime="1.681125982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:49:45.752580821 +0000 UTC m=+287.401513364" watchObservedRunningTime="2026-06-10 10:49:45.754414428 +0000 UTC m=+287.403347008" Jun 10 10:49:55.741232 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:55.741196 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tc8c9" Jun 10 10:49:58.734506 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:49:58.734482 2545 kubelet.go:1648] "Image garbage collection succeeded" Jun 10 10:50:15.101233 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.101198 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:50:15.103900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.101492 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="alertmanager" containerID="cri-o://be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4" gracePeriod=120 Jun 10 10:50:15.103900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.101544 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-metric" containerID="cri-o://d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f" gracePeriod=120 Jun 10 10:50:15.103900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.101596 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-web" containerID="cri-o://b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6" gracePeriod=120 Jun 10 10:50:15.103900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.101588 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy" containerID="cri-o://de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091" gracePeriod=120 Jun 10 10:50:15.103900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.101641 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="prom-label-proxy" containerID="cri-o://baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e" gracePeriod=120 Jun 10 10:50:15.103900 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.101596 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="config-reloader" containerID="cri-o://5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e" gracePeriod=120 Jun 10 10:50:15.845713 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845678 2545 generic.go:356] "Generic (PLEG): container finished" podID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerID="baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e" exitCode=0 Jun 10 10:50:15.845713 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845706 2545 generic.go:356] "Generic (PLEG): container finished" podID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerID="d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f" exitCode=0 Jun 10 10:50:15.845713 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845714 2545 generic.go:356] "Generic (PLEG): container finished" podID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerID="de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091" exitCode=0 Jun 10 10:50:15.845713 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845721 2545 generic.go:356] "Generic (PLEG): container finished" podID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerID="5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e" exitCode=0 Jun 10 10:50:15.845713 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845728 2545 generic.go:356] "Generic (PLEG): container finished" podID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerID="be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4" exitCode=0 Jun 10 10:50:15.846029 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845742 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e"} Jun 10 10:50:15.846029 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845806 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f"} Jun 10 10:50:15.846029 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845832 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091"} Jun 10 10:50:15.846029 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845857 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e"} Jun 10 10:50:15.846029 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:15.845882 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4"} Jun 10 10:50:16.346317 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.346290 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.429459 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429426 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9hts\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-kube-api-access-t9hts\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.429644 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429479 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.429644 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429500 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.429644 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429537 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-tls-assets\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.429644 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429560 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-trusted-ca-bundle\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.429644 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429577 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-web-config\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.429644 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429592 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-cluster-tls-config\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.429644 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429617 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-metric\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.430012 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429650 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-config-volume\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.430012 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429683 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-metrics-client-ca\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.430012 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429709 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-main-db\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.430012 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429736 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-web\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.430012 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.429770 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-config-out\") pod \"bd396cfd-5d99-425b-ad86-36052cfd5035\" (UID: \"bd396cfd-5d99-425b-ad86-36052cfd5035\") " Jun 10 10:50:16.430296 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.430216 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:50:16.430415 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.430388 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 10:50:16.430813 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.430755 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 10:50:16.432384 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.432346 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:16.432565 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.432537 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:50:16.432565 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.432550 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-config-out" (OuterVolumeSpecName: "config-out") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 10:50:16.432731 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.432629 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:16.432731 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.432703 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-kube-api-access-t9hts" (OuterVolumeSpecName: "kube-api-access-t9hts") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "kube-api-access-t9hts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:50:16.433483 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.433440 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:16.434111 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.434053 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:16.434111 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.434096 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-config-volume" (OuterVolumeSpecName: "config-volume") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:16.436672 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.436550 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:16.442444 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.442420 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-web-config" (OuterVolumeSpecName: "web-config") pod "bd396cfd-5d99-425b-ad86-36052cfd5035" (UID: "bd396cfd-5d99-425b-ad86-36052cfd5035"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 10:50:16.531025 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.530950 2545 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531025 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531018 2545 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-main-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531025 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531031 2545 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-tls-assets\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531025 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531040 2545 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531049 2545 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-web-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531058 2545 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-cluster-tls-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531066 2545 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531074 2545 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531085 2545 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd396cfd-5d99-425b-ad86-36052cfd5035-metrics-client-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531092 2545 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-alertmanager-main-db\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531100 2545 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd396cfd-5d99-425b-ad86-36052cfd5035-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531109 2545 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd396cfd-5d99-425b-ad86-36052cfd5035-config-out\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.531290 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.531117 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hts\" (UniqueName: \"kubernetes.io/projected/bd396cfd-5d99-425b-ad86-36052cfd5035-kube-api-access-t9hts\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:50:16.851706 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.851670 2545 generic.go:356] "Generic (PLEG): container finished" podID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerID="b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6" exitCode=0 Jun 10 10:50:16.851882 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.851723 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6"} Jun 10 10:50:16.852095 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.852069 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd396cfd-5d99-425b-ad86-36052cfd5035","Type":"ContainerDied","Data":"d2b48154a5ef60e884a3692eb0e41163d32ce11606c55404d6257fdeafef3221"} Jun 10 10:50:16.852182 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.852098 2545 scope.go:117] "RemoveContainer" containerID="baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e" Jun 10 10:50:16.856224 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.852875 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.863436 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.863419 2545 scope.go:117] "RemoveContainer" containerID="d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f" Jun 10 10:50:16.870128 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.870103 2545 scope.go:117] "RemoveContainer" containerID="de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091" Jun 10 10:50:16.876785 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.876766 2545 scope.go:117] "RemoveContainer" containerID="b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6" Jun 10 10:50:16.877221 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.877082 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:50:16.881980 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.881947 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:50:16.883718 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.883697 2545 scope.go:117] "RemoveContainer" containerID="5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e" Jun 10 10:50:16.890342 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.890327 2545 scope.go:117] "RemoveContainer" containerID="be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4" Jun 10 10:50:16.897039 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.897020 2545 scope.go:117] "RemoveContainer" containerID="f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e" Jun 10 10:50:16.903561 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.903545 2545 scope.go:117] "RemoveContainer" containerID="baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e" Jun 10 10:50:16.903837 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:50:16.903820 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e\": container with ID starting with baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e not found: ID does not exist" containerID="baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e" Jun 10 10:50:16.903879 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.903847 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e"} err="failed to get container status \"baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e\": rpc error: code = NotFound desc = could not find container \"baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e\": container with ID starting with baf1e93dd1184cf883d66923faec77791cdad097d924026cd6dc004b34a79a2e not found: ID does not exist" Jun 10 10:50:16.903879 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.903865 2545 scope.go:117] "RemoveContainer" containerID="d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f" Jun 10 10:50:16.904110 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:50:16.904094 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f\": container with ID starting with d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f not found: ID does not exist" containerID="d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904114 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f"} err="failed to get container status \"d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f\": rpc error: code = NotFound desc = could not find container \"d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f\": container with ID starting with d4753e755ad00d1e98bec25d9861cede597acb9ad59f80856a5f338e803be75f not found: ID does not exist" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904129 2545 scope.go:117] "RemoveContainer" containerID="de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:50:16.904339 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091\": container with ID starting with de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091 not found: ID does not exist" containerID="de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904361 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091"} err="failed to get container status \"de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091\": rpc error: code = NotFound desc = could not find container \"de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091\": container with ID starting with de8e4215300d171efa12144695d6da0a82f1e51a001c2d8934b75a71f49c5091 not found: ID does not exist" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904376 2545 scope.go:117] "RemoveContainer" containerID="b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:50:16.904563 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6\": container with ID starting with b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6 not found: ID does not exist" containerID="b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904577 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6"} err="failed to get container status \"b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6\": rpc error: code = NotFound desc = could not find container \"b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6\": container with ID starting with b922dab623b2b15b7c0d7ebb6dcaef37547acff292e619a2499fe127ed1423b6 not found: ID does not exist" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904589 2545 scope.go:117] "RemoveContainer" containerID="5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:50:16.904753 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e\": container with ID starting with 5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e not found: ID does not exist" containerID="5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904766 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e"} err="failed to get container status \"5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e\": rpc error: code = NotFound desc = could not find container \"5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e\": container with ID starting with 5dae901bf34ca2477349ac57f672288948cbb5201f2f9c7abe0cfcd85c31ec1e not found: ID does not exist" Jun 10 10:50:16.904842 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.904776 2545 scope.go:117] "RemoveContainer" containerID="be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4" Jun 10 10:50:16.905258 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:50:16.905025 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4\": container with ID starting with be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4 not found: ID does not exist" containerID="be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4" Jun 10 10:50:16.905258 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.905047 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4"} err="failed to get container status \"be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4\": rpc error: code = NotFound desc = could not find container \"be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4\": container with ID starting with be23f9a1fe83753efbdcbfc5b50855e4d3d1d77811d9fa99d7f291c409055cd4 not found: ID does not exist" Jun 10 10:50:16.905258 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.905059 2545 scope.go:117] "RemoveContainer" containerID="f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e" Jun 10 10:50:16.905373 ip-10-0-128-191 kubenswrapper[2545]: E0610 10:50:16.905276 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e\": container with ID starting with f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e not found: ID does not exist" containerID="f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e" Jun 10 10:50:16.905373 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.905305 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e"} err="failed to get container status \"f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e\": rpc error: code = NotFound desc = could not find container \"f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e\": container with ID starting with f0c6bc544a6817dd8b7e18c67755def9ba1bc67362b87ef3dbd43a668036df3e not found: ID does not exist" Jun 10 10:50:16.909603 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909581 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:50:16.909846 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909835 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="config-reloader" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909848 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="config-reloader" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909858 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-metric" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909863 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-metric" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909871 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" containerName="console" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909876 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" containerName="console" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909883 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="prom-label-proxy" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909888 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="prom-label-proxy" Jun 10 10:50:16.909891 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909894 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="alertmanager" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909899 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="alertmanager" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909904 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-web" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909910 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-web" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909919 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="init-config-reloader" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909923 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="init-config-reloader" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909929 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909934 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.909996 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="alertmanager" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.910003 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.910010 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-web" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.910015 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="prom-label-proxy" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.910021 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="a02b0a61-0a66-4c9b-9f76-1aa3fbcee8a9" containerName="console" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.910025 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="config-reloader" Jun 10 10:50:16.910154 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.910030 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" containerName="kube-rbac-proxy-metric" Jun 10 10:50:16.915121 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.915100 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.917811 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.917790 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 10 10:50:16.918145 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918121 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 10 10:50:16.918252 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918163 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 10 10:50:16.918309 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918288 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 10 10:50:16.918414 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918395 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 10 10:50:16.918466 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918411 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 10 10:50:16.918466 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918441 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 10 10:50:16.918561 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918534 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 10 10:50:16.918669 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.918654 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6bl24\"" Jun 10 10:50:16.923874 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.923854 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 10 10:50:16.927703 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.927681 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:50:16.934859 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.934839 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.934859 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.934866 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36462199-1c62-4df5-bb91-c7b0b58cdce4-config-out\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935011 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.934885 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-web-config\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935011 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.934933 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttsh\" (UniqueName: \"kubernetes.io/projected/36462199-1c62-4df5-bb91-c7b0b58cdce4-kube-api-access-tttsh\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935011 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935002 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36462199-1c62-4df5-bb91-c7b0b58cdce4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935108 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935022 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935108 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935042 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36462199-1c62-4df5-bb91-c7b0b58cdce4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935108 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935100 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935188 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935117 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935188 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935134 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36462199-1c62-4df5-bb91-c7b0b58cdce4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935188 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935175 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935278 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935205 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-config-volume\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:16.935278 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:16.935222 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36462199-1c62-4df5-bb91-c7b0b58cdce4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.035618 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.035574 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36462199-1c62-4df5-bb91-c7b0b58cdce4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.035618 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.035621 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.035835 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.035749 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-config-volume\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.035835 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.035798 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36462199-1c62-4df5-bb91-c7b0b58cdce4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.035897 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.035844 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.035897 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.035872 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36462199-1c62-4df5-bb91-c7b0b58cdce4-config-out\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036043 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036023 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-web-config\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036101 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036076 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttsh\" (UniqueName: \"kubernetes.io/projected/36462199-1c62-4df5-bb91-c7b0b58cdce4-kube-api-access-tttsh\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036148 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036113 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36462199-1c62-4df5-bb91-c7b0b58cdce4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036201 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036144 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036201 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036185 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36462199-1c62-4df5-bb91-c7b0b58cdce4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036227 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036299 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036252 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036601 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036568 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36462199-1c62-4df5-bb91-c7b0b58cdce4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036849 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036825 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36462199-1c62-4df5-bb91-c7b0b58cdce4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.036932 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.036832 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36462199-1c62-4df5-bb91-c7b0b58cdce4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.038722 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.038640 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36462199-1c62-4df5-bb91-c7b0b58cdce4-config-out\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.038820 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.038726 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.038921 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.038887 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.039003 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.038889 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-config-volume\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.039217 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.039192 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.039296 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.039237 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-web-config\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.039461 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.039442 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36462199-1c62-4df5-bb91-c7b0b58cdce4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.039676 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.039658 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.040416 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.040400 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36462199-1c62-4df5-bb91-c7b0b58cdce4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.046685 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.046667 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttsh\" (UniqueName: \"kubernetes.io/projected/36462199-1c62-4df5-bb91-c7b0b58cdce4-kube-api-access-tttsh\") pod \"alertmanager-main-0\" (UID: \"36462199-1c62-4df5-bb91-c7b0b58cdce4\") " pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.225263 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.225223 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 10 10:50:17.361935 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.361909 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 10 10:50:17.363778 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:50:17.363754 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36462199_1c62_4df5_bb91_c7b0b58cdce4.slice/crio-bc950ffc987cde4e257e45b3cab7c3ebd25e320cf37269591694f05de24dac47 WatchSource:0}: Error finding container bc950ffc987cde4e257e45b3cab7c3ebd25e320cf37269591694f05de24dac47: Status 404 returned error can't find the container with id bc950ffc987cde4e257e45b3cab7c3ebd25e320cf37269591694f05de24dac47 Jun 10 10:50:17.857006 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.856946 2545 generic.go:356] "Generic (PLEG): container finished" podID="36462199-1c62-4df5-bb91-c7b0b58cdce4" containerID="96ea15f05916daf9cc15c140ec5979f9c69a9a392d9d7ddfd7d46a0989ca74e3" exitCode=0 Jun 10 10:50:17.857209 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.857031 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerDied","Data":"96ea15f05916daf9cc15c140ec5979f9c69a9a392d9d7ddfd7d46a0989ca74e3"} Jun 10 10:50:17.857209 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:17.857108 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerStarted","Data":"bc950ffc987cde4e257e45b3cab7c3ebd25e320cf37269591694f05de24dac47"} Jun 10 10:50:18.782332 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.782301 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd396cfd-5d99-425b-ad86-36052cfd5035" path="/var/lib/kubelet/pods/bd396cfd-5d99-425b-ad86-36052cfd5035/volumes" Jun 10 10:50:18.864955 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.864919 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerStarted","Data":"29c5eda4f1ae9fcbe7baa43acb95bd9c663a558c89843ef698ac3854510d75d5"} Jun 10 10:50:18.865109 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.865017 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerStarted","Data":"59cbc783622c63f10a2823cc5cb966c2869a1ef49c4ccd4bee7cb1fb09505d05"} Jun 10 10:50:18.865109 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.865062 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerStarted","Data":"8469ae0c392b3e29a326ad4d3a74df9db4c5312e52a558598ddc58310a394af3"} Jun 10 10:50:18.865109 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.865097 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerStarted","Data":"77046024b1782e97814a2f40ccdc765769fbb08b8f35ecdb47ca0bc8707b449d"} Jun 10 10:50:18.865217 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.865130 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerStarted","Data":"0ca8907f332fa896ef9f34205f4264d50449d6a124802b7218d8428ec214b64f"} Jun 10 10:50:18.865217 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.865169 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36462199-1c62-4df5-bb91-c7b0b58cdce4","Type":"ContainerStarted","Data":"6a36deeeff2486be0ed25883a1a6613918b7acf64b36c7ab3e04df8923fc6855"} Jun 10 10:50:18.895939 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:50:18.895879 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.880153685 podStartE2EDuration="2.895858819s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.705134ms" isStatefulPod=false podCreationTimestamp="2026-06-10 10:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:50:18.89401171 +0000 UTC m=+320.542944250" watchObservedRunningTime="2026-06-10 10:50:18.895858819 +0000 UTC m=+320.544791364" Jun 10 10:54:12.257242 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.257159 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-596f78d89b-89nfr"] Jun 10 10:54:12.260478 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.260455 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.263531 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.263498 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jun 10 10:54:12.263531 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.263498 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qzwfh\"" Jun 10 10:54:12.264913 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.264892 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jun 10 10:54:12.264913 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.264903 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jun 10 10:54:12.265099 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.264958 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jun 10 10:54:12.265155 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.265136 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jun 10 10:54:12.270226 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.270204 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jun 10 10:54:12.277748 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.277722 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596f78d89b-89nfr"] Jun 10 10:54:12.421223 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.421179 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-service-ca\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.421223 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.421232 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54075fc2-edfe-4638-bb25-4988461b43b1-console-serving-cert\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.421460 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.421257 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-oauth-serving-cert\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.421460 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.421308 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-trusted-ca-bundle\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.421460 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.421345 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54075fc2-edfe-4638-bb25-4988461b43b1-console-oauth-config\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.421460 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.421412 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bcb\" (UniqueName: \"kubernetes.io/projected/54075fc2-edfe-4638-bb25-4988461b43b1-kube-api-access-z7bcb\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.421460 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.421441 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-console-config\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.522637 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.522540 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-service-ca\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.522637 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.522586 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54075fc2-edfe-4638-bb25-4988461b43b1-console-serving-cert\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.522637 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.522615 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-oauth-serving-cert\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.522637 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.522638 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-trusted-ca-bundle\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.522947 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.522656 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54075fc2-edfe-4638-bb25-4988461b43b1-console-oauth-config\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.522947 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.522700 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bcb\" (UniqueName: \"kubernetes.io/projected/54075fc2-edfe-4638-bb25-4988461b43b1-kube-api-access-z7bcb\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.522947 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.522733 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-console-config\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.523473 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.523448 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-oauth-serving-cert\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.523598 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.523566 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-trusted-ca-bundle\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.523598 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.523537 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-service-ca\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.523689 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.523459 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54075fc2-edfe-4638-bb25-4988461b43b1-console-config\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.525119 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.525102 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54075fc2-edfe-4638-bb25-4988461b43b1-console-serving-cert\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.525220 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.525202 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54075fc2-edfe-4638-bb25-4988461b43b1-console-oauth-config\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.532486 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.532460 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bcb\" (UniqueName: \"kubernetes.io/projected/54075fc2-edfe-4638-bb25-4988461b43b1-kube-api-access-z7bcb\") pod \"console-596f78d89b-89nfr\" (UID: \"54075fc2-edfe-4638-bb25-4988461b43b1\") " pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.571396 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.571356 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:12.709178 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:12.709147 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596f78d89b-89nfr"] Jun 10 10:54:12.712169 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:54:12.712142 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54075fc2_edfe_4638_bb25_4988461b43b1.slice/crio-52a8ae570f65aeb4911dd8d288cf38d41b9c0d6f5612fdb00790079f1014551a WatchSource:0}: Error finding container 52a8ae570f65aeb4911dd8d288cf38d41b9c0d6f5612fdb00790079f1014551a: Status 404 returned error can't find the container with id 52a8ae570f65aeb4911dd8d288cf38d41b9c0d6f5612fdb00790079f1014551a Jun 10 10:54:13.686488 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:13.686448 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596f78d89b-89nfr" event={"ID":"54075fc2-edfe-4638-bb25-4988461b43b1","Type":"ContainerStarted","Data":"c5c153efb22b56f2badb5586f2eb782a13a962274913b395d4534a62cc50774f"} Jun 10 10:54:13.686859 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:13.686532 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596f78d89b-89nfr" event={"ID":"54075fc2-edfe-4638-bb25-4988461b43b1","Type":"ContainerStarted","Data":"52a8ae570f65aeb4911dd8d288cf38d41b9c0d6f5612fdb00790079f1014551a"} Jun 10 10:54:13.706726 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:13.706669 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-596f78d89b-89nfr" podStartSLOduration=1.7066535809999999 podStartE2EDuration="1.706653581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:54:13.705523792 +0000 UTC m=+555.354456338" watchObservedRunningTime="2026-06-10 10:54:13.706653581 +0000 UTC m=+555.355586121" Jun 10 10:54:22.572295 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:22.572247 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:22.572746 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:22.572346 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:22.577218 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:22.577193 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:54:22.727425 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:54:22.727398 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-596f78d89b-89nfr" Jun 10 10:59:03.852354 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.852315 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-test-apply-nvxfl"] Jun 10 10:59:03.855381 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.855363 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-nvxfl" Jun 10 10:59:03.858622 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.858590 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-zsxr2\"" Jun 10 10:59:03.858622 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.858590 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jun 10 10:59:03.859843 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.859824 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 10 10:59:03.859947 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.859848 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 10 10:59:03.865323 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.865299 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-nvxfl"] Jun 10 10:59:03.937898 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:03.937856 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8dn\" (UniqueName: \"kubernetes.io/projected/3bf8585a-0629-4d06-9723-1ea18069f7d5-kube-api-access-2j8dn\") pod \"feast-test-apply-nvxfl\" (UID: \"3bf8585a-0629-4d06-9723-1ea18069f7d5\") " pod="test-ns-feast/feast-test-apply-nvxfl" Jun 10 10:59:04.038701 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:04.038672 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8dn\" (UniqueName: \"kubernetes.io/projected/3bf8585a-0629-4d06-9723-1ea18069f7d5-kube-api-access-2j8dn\") pod \"feast-test-apply-nvxfl\" (UID: \"3bf8585a-0629-4d06-9723-1ea18069f7d5\") " pod="test-ns-feast/feast-test-apply-nvxfl" Jun 10 10:59:04.049596 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:04.049561 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8dn\" (UniqueName: \"kubernetes.io/projected/3bf8585a-0629-4d06-9723-1ea18069f7d5-kube-api-access-2j8dn\") pod \"feast-test-apply-nvxfl\" (UID: \"3bf8585a-0629-4d06-9723-1ea18069f7d5\") " pod="test-ns-feast/feast-test-apply-nvxfl" Jun 10 10:59:04.165177 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:04.165128 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-nvxfl" Jun 10 10:59:04.299577 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:04.299506 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-test-apply-nvxfl"] Jun 10 10:59:04.302143 ip-10-0-128-191 kubenswrapper[2545]: W0610 10:59:04.302115 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf8585a_0629_4d06_9723_1ea18069f7d5.slice/crio-205c7b65a6ffeeb6c0a0598d903c1ecf26aab093986eb89afe4a052e1022d7b8 WatchSource:0}: Error finding container 205c7b65a6ffeeb6c0a0598d903c1ecf26aab093986eb89afe4a052e1022d7b8: Status 404 returned error can't find the container with id 205c7b65a6ffeeb6c0a0598d903c1ecf26aab093986eb89afe4a052e1022d7b8 Jun 10 10:59:04.303915 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:04.303901 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 10 10:59:04.724482 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:04.724445 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-nvxfl" event={"ID":"3bf8585a-0629-4d06-9723-1ea18069f7d5","Type":"ContainerStarted","Data":"205c7b65a6ffeeb6c0a0598d903c1ecf26aab093986eb89afe4a052e1022d7b8"} Jun 10 10:59:09.745265 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:09.745171 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-nvxfl" event={"ID":"3bf8585a-0629-4d06-9723-1ea18069f7d5","Type":"ContainerStarted","Data":"bafb1e782c1e04c910c205e8d17c34bea2576fb2b9374e15a979e0db2cd93c57"} Jun 10 10:59:15.775145 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:15.775107 2545 generic.go:356] "Generic (PLEG): container finished" podID="3bf8585a-0629-4d06-9723-1ea18069f7d5" containerID="bafb1e782c1e04c910c205e8d17c34bea2576fb2b9374e15a979e0db2cd93c57" exitCode=0 Jun 10 10:59:15.775624 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:15.775173 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-nvxfl" event={"ID":"3bf8585a-0629-4d06-9723-1ea18069f7d5","Type":"ContainerDied","Data":"bafb1e782c1e04c910c205e8d17c34bea2576fb2b9374e15a979e0db2cd93c57"} Jun 10 10:59:16.781292 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:16.781259 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-nvxfl" event={"ID":"3bf8585a-0629-4d06-9723-1ea18069f7d5","Type":"ContainerStarted","Data":"0ba5c86ab6a8187ccb091446d53f024d32b54a388b22b5c654a7caea5f947169"} Jun 10 10:59:16.807241 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:16.807191 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-test-apply-nvxfl" podStartSLOduration=2.444093854 podStartE2EDuration="13.807175102s" totalImagesPullingTime="5.186865827s" totalInitContainerRuntime="6.176215421s" isStatefulPod=false podCreationTimestamp="2026-06-10 10:59:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 10:59:16.806896951 +0000 UTC m=+858.455829505" watchObservedRunningTime="2026-06-10 10:59:16.807175102 +0000 UTC m=+858.456107643" Jun 10 10:59:25.815396 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:25.815363 2545 generic.go:356] "Generic (PLEG): container finished" podID="3bf8585a-0629-4d06-9723-1ea18069f7d5" containerID="0ba5c86ab6a8187ccb091446d53f024d32b54a388b22b5c654a7caea5f947169" exitCode=0 Jun 10 10:59:25.815784 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:25.815434 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-nvxfl" event={"ID":"3bf8585a-0629-4d06-9723-1ea18069f7d5","Type":"ContainerDied","Data":"0ba5c86ab6a8187ccb091446d53f024d32b54a388b22b5c654a7caea5f947169"} Jun 10 10:59:26.955131 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:26.955100 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-nvxfl" Jun 10 10:59:27.027305 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:27.027267 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j8dn\" (UniqueName: \"kubernetes.io/projected/3bf8585a-0629-4d06-9723-1ea18069f7d5-kube-api-access-2j8dn\") pod \"3bf8585a-0629-4d06-9723-1ea18069f7d5\" (UID: \"3bf8585a-0629-4d06-9723-1ea18069f7d5\") " Jun 10 10:59:27.029536 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:27.029505 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf8585a-0629-4d06-9723-1ea18069f7d5-kube-api-access-2j8dn" (OuterVolumeSpecName: "kube-api-access-2j8dn") pod "3bf8585a-0629-4d06-9723-1ea18069f7d5" (UID: "3bf8585a-0629-4d06-9723-1ea18069f7d5"). InnerVolumeSpecName "kube-api-access-2j8dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 10:59:27.128741 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:27.128700 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8dn\" (UniqueName: \"kubernetes.io/projected/3bf8585a-0629-4d06-9723-1ea18069f7d5-kube-api-access-2j8dn\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 10:59:27.825462 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:27.825422 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-test-apply-nvxfl" event={"ID":"3bf8585a-0629-4d06-9723-1ea18069f7d5","Type":"ContainerDied","Data":"205c7b65a6ffeeb6c0a0598d903c1ecf26aab093986eb89afe4a052e1022d7b8"} Jun 10 10:59:27.825462 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:27.825444 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-test-apply-nvxfl" Jun 10 10:59:27.825462 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:27.825470 2545 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="205c7b65a6ffeeb6c0a0598d903c1ecf26aab093986eb89afe4a052e1022d7b8" Jun 10 10:59:28.037280 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:28.037255 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-nvxfl_3bf8585a-0629-4d06-9723-1ea18069f7d5/feast-0/0.log" Jun 10 10:59:28.046837 ip-10-0-128-191 kubenswrapper[2545]: I0610 10:59:28.046811 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/test-ns-feast_feast-test-apply-nvxfl_3bf8585a-0629-4d06-9723-1ea18069f7d5/feast-1/0.log" Jun 10 11:00:02.880550 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:00:02.880469 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-test-apply-nvxfl"] Jun 10 11:00:02.884947 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:00:02.884919 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-test-apply-nvxfl"] Jun 10 11:00:04.781745 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:00:04.781713 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf8585a-0629-4d06-9723-1ea18069f7d5" path="/var/lib/kubelet/pods/3bf8585a-0629-4d06-9723-1ea18069f7d5/volumes" Jun 10 11:04:44.469640 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.469602 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2"] Jun 10 11:04:44.470093 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.469870 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf8585a-0629-4d06-9723-1ea18069f7d5" containerName="feast-0" Jun 10 11:04:44.470093 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.469881 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf8585a-0629-4d06-9723-1ea18069f7d5" containerName="feast-0" Jun 10 11:04:44.470093 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.469895 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf8585a-0629-4d06-9723-1ea18069f7d5" containerName="feast-1" Jun 10 11:04:44.470093 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.469901 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf8585a-0629-4d06-9723-1ea18069f7d5" containerName="feast-1" Jun 10 11:04:44.470093 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.469957 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf8585a-0629-4d06-9723-1ea18069f7d5" containerName="feast-1" Jun 10 11:04:44.472864 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.472847 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:04:44.476908 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.476882 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jun 10 11:04:44.478118 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.478095 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jun 10 11:04:44.478223 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.478101 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-stjlw\"" Jun 10 11:04:44.490388 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.490359 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2"] Jun 10 11:04:44.595609 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.595558 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dsl\" (UniqueName: \"kubernetes.io/projected/2617864f-c265-4f54-8003-d3d75b23d2d3-kube-api-access-t5dsl\") pod \"feast-operator-controller-manager-6d54c7db9d-lktj2\" (UID: \"2617864f-c265-4f54-8003-d3d75b23d2d3\") " pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:04:44.696245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.696205 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dsl\" (UniqueName: \"kubernetes.io/projected/2617864f-c265-4f54-8003-d3d75b23d2d3-kube-api-access-t5dsl\") pod \"feast-operator-controller-manager-6d54c7db9d-lktj2\" (UID: \"2617864f-c265-4f54-8003-d3d75b23d2d3\") " pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:04:44.707416 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.707383 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dsl\" (UniqueName: \"kubernetes.io/projected/2617864f-c265-4f54-8003-d3d75b23d2d3-kube-api-access-t5dsl\") pod \"feast-operator-controller-manager-6d54c7db9d-lktj2\" (UID: \"2617864f-c265-4f54-8003-d3d75b23d2d3\") " pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:04:44.783215 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.783127 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:04:44.918517 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.918482 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2"] Jun 10 11:04:44.921711 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:04:44.921673 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2617864f_c265_4f54_8003_d3d75b23d2d3.slice/crio-f66f40c2f319c16d0aa160eef2af7168076ff03d39ad82430441dffe955a5dbe WatchSource:0}: Error finding container f66f40c2f319c16d0aa160eef2af7168076ff03d39ad82430441dffe955a5dbe: Status 404 returned error can't find the container with id f66f40c2f319c16d0aa160eef2af7168076ff03d39ad82430441dffe955a5dbe Jun 10 11:04:44.923566 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.923550 2545 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 10 11:04:44.973574 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:44.973534 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" event={"ID":"2617864f-c265-4f54-8003-d3d75b23d2d3","Type":"ContainerStarted","Data":"f66f40c2f319c16d0aa160eef2af7168076ff03d39ad82430441dffe955a5dbe"} Jun 10 11:04:47.989802 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:47.989759 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" event={"ID":"2617864f-c265-4f54-8003-d3d75b23d2d3","Type":"ContainerStarted","Data":"5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0"} Jun 10 11:04:47.990228 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:47.989864 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:04:58.996175 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:58.996146 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:04:59.032108 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:04:59.032058 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" podStartSLOduration=12.140334577 podStartE2EDuration="15.032041104s" totalImagesPullingTime="2.891706527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:04:48.016457642 +0000 UTC m=+1189.665390184" watchObservedRunningTime="2026-06-10 11:04:59.032041104 +0000 UTC m=+1200.680973645" Jun 10 11:05:02.775227 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.775184 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh"] Jun 10 11:05:02.778976 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.778940 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:02.796221 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.796191 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 10 11:05:02.796415 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.796295 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 10 11:05:02.796415 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.796327 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 10 11:05:02.798974 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.798944 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 10 11:05:02.799121 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.799079 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 10 11:05:02.799216 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.799202 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 10 11:05:02.799348 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.799334 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-sqtlt\"" Jun 10 11:05:02.825011 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.824982 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh"] Jun 10 11:05:02.942600 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.942565 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:02.942792 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.942705 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:02.942792 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.942745 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:02.942792 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.942768 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl64p\" (UniqueName: \"kubernetes.io/projected/1f152be3-1b81-4121-8213-093d22a25d99-kube-api-access-wl64p\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:02.942889 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.942824 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f152be3-1b81-4121-8213-093d22a25d99-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:02.942889 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:02.942858 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.043326 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.043234 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.043326 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.043275 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.043326 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.043300 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl64p\" (UniqueName: \"kubernetes.io/projected/1f152be3-1b81-4121-8213-093d22a25d99-kube-api-access-wl64p\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.043589 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.043468 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f152be3-1b81-4121-8213-093d22a25d99-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.043589 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.043505 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.043589 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.043558 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.043906 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.043878 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f152be3-1b81-4121-8213-093d22a25d99-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.045994 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.045954 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.046180 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.046155 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.046281 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.046156 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.046281 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.046252 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.054435 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.054408 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl64p\" (UniqueName: \"kubernetes.io/projected/1f152be3-1b81-4121-8213-093d22a25d99-kube-api-access-wl64p\") pod \"feast-simple-feast-setup-75c64c888b-gchnh\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.093216 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.093180 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:03.259831 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:03.259800 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh"] Jun 10 11:05:03.263150 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:05:03.263122 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f152be3_1b81_4121_8213_093d22a25d99.slice/crio-c645e5d972696722c881b87cedce37cd4da76e6b670c2b6903981167cccc5a66 WatchSource:0}: Error finding container c645e5d972696722c881b87cedce37cd4da76e6b670c2b6903981167cccc5a66: Status 404 returned error can't find the container with id c645e5d972696722c881b87cedce37cd4da76e6b670c2b6903981167cccc5a66 Jun 10 11:05:04.054842 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:04.054794 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerStarted","Data":"c645e5d972696722c881b87cedce37cd4da76e6b670c2b6903981167cccc5a66"} Jun 10 11:05:19.124940 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:19.124894 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerStarted","Data":"d3545e90fcf9710099f2aa4ee41bff2dc6f2e8371cf7a9b532c5c81abf227678"} Jun 10 11:05:24.145822 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:24.145787 2545 generic.go:356] "Generic (PLEG): container finished" podID="1f152be3-1b81-4121-8213-093d22a25d99" containerID="d3545e90fcf9710099f2aa4ee41bff2dc6f2e8371cf7a9b532c5c81abf227678" exitCode=0 Jun 10 11:05:24.146236 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:24.145849 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerDied","Data":"d3545e90fcf9710099f2aa4ee41bff2dc6f2e8371cf7a9b532c5c81abf227678"} Jun 10 11:05:25.151987 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:25.151937 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerStarted","Data":"c2fb43952178408d1e71467d18a3eb67da71377e0305d7d4dcdc69db5c83620e"} Jun 10 11:05:31.178824 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:31.178791 2545 generic.go:356] "Generic (PLEG): container finished" podID="1f152be3-1b81-4121-8213-093d22a25d99" containerID="c2fb43952178408d1e71467d18a3eb67da71377e0305d7d4dcdc69db5c83620e" exitCode=0 Jun 10 11:05:31.179270 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:31.178868 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerDied","Data":"c2fb43952178408d1e71467d18a3eb67da71377e0305d7d4dcdc69db5c83620e"} Jun 10 11:05:32.189456 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:32.189406 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerStarted","Data":"06ab8c77a04542206af82ebd0147b8af323059af28aa00e8a45656a9b2c37c7e"} Jun 10 11:05:32.189955 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:32.189488 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerStarted","Data":"270a4d12870bb37fbbb5c3b6b0ec40fc5239e68810a073f67053e06191071661"} Jun 10 11:05:32.189955 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:32.189511 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerStarted","Data":"97a48d9b3ad4e1a8773b482b3c806ac152c3cad8e0ed7072a59173ee50f2fabd"} Jun 10 11:05:32.189955 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:32.189529 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerStarted","Data":"d7eccb19b155785a515e23a210af4288b98439ba2053813dffa194cd07fd6cf6"} Jun 10 11:05:32.222885 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:32.222812 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podStartSLOduration=15.017766735 podStartE2EDuration="30.222786884s" totalImagesPullingTime="15.205020149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:05:32.220424231 +0000 UTC m=+1233.869356801" watchObservedRunningTime="2026-06-10 11:05:32.222786884 +0000 UTC m=+1233.871719427" Jun 10 11:05:33.094138 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094097 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.094424 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094406 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.094542 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094530 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.094657 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094633 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.094657 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094662 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.094834 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094683 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.094834 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094701 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.094834 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.094718 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:33.096615 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.096574 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jun 10 11:05:33.096754 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.096606 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.20:8816: connect: connection refused" Jun 10 11:05:33.096754 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.096637 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.20:6571: connect: connection refused" Jun 10 11:05:33.096869 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:33.096789 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jun 10 11:05:36.094322 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:36.094178 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jun 10 11:05:36.094322 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:36.094242 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jun 10 11:05:36.094322 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:36.094178 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.20:6571: connect: connection refused" Jun 10 11:05:36.094322 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:36.094177 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.20:8816: connect: connection refused" Jun 10 11:05:39.094645 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.094607 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:39.097323 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.094884 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:39.097323 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.095249 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:39.097323 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.095310 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:39.099311 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.099288 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:39.224154 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.224117 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:39.224358 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.224201 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:39.226813 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:39.226795 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:05:58.821213 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:58.821181 2545 scope.go:117] "RemoveContainer" containerID="0ba5c86ab6a8187ccb091446d53f024d32b54a388b22b5c654a7caea5f947169" Jun 10 11:05:58.828807 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:05:58.828780 2545 scope.go:117] "RemoveContainer" containerID="bafb1e782c1e04c910c205e8d17c34bea2576fb2b9374e15a979e0db2cd93c57" Jun 10 11:06:40.478008 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.477956 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh"] Jun 10 11:06:40.478611 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.478230 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="registry" containerID="cri-o://d7eccb19b155785a515e23a210af4288b98439ba2053813dffa194cd07fd6cf6" gracePeriod=30 Jun 10 11:06:40.478611 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.478250 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="offline" containerID="cri-o://270a4d12870bb37fbbb5c3b6b0ec40fc5239e68810a073f67053e06191071661" gracePeriod=30 Jun 10 11:06:40.478611 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.478262 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" containerID="cri-o://06ab8c77a04542206af82ebd0147b8af323059af28aa00e8a45656a9b2c37c7e" gracePeriod=30 Jun 10 11:06:40.478611 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.478303 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" containerID="cri-o://97a48d9b3ad4e1a8773b482b3c806ac152c3cad8e0ed7072a59173ee50f2fabd" gracePeriod=30 Jun 10 11:06:40.814638 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.814539 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8"] Jun 10 11:06:40.817453 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.817425 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:40.821161 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.821137 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-m4f9k\"" Jun 10 11:06:40.834771 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.834737 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8"] Jun 10 11:06:40.966712 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.966678 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:40.966712 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.966717 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvz9\" (UniqueName: \"kubernetes.io/projected/bbc92800-db0b-4153-b564-bdc84f06a209-kube-api-access-8pvz9\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:40.966983 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.966739 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc92800-db0b-4153-b564-bdc84f06a209-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:40.966983 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.966759 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:40.966983 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.966943 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:40.967125 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:40.967016 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.068357 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.068265 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.068357 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.068310 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.068584 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.068452 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.068584 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.068508 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvz9\" (UniqueName: \"kubernetes.io/projected/bbc92800-db0b-4153-b564-bdc84f06a209-kube-api-access-8pvz9\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.068584 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.068544 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc92800-db0b-4153-b564-bdc84f06a209-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.068744 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.068585 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.069084 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.069059 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc92800-db0b-4153-b564-bdc84f06a209-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.071244 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.071220 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.071346 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.071220 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.071346 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.071280 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.071542 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.071526 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.079864 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.079832 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvz9\" (UniqueName: \"kubernetes.io/projected/bbc92800-db0b-4153-b564-bdc84f06a209-kube-api-access-8pvz9\") pod \"feast-simple-feast-setup-75c64c888b-t5kf8\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.128146 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.128094 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:41.294427 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.294401 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8"] Jun 10 11:06:41.296721 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:06:41.296693 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc92800_db0b_4153_b564_bdc84f06a209.slice/crio-ca910092d78153e8d9b5e5be2b04f0651649e2a2623725cf20817cb897817d5e WatchSource:0}: Error finding container ca910092d78153e8d9b5e5be2b04f0651649e2a2623725cf20817cb897817d5e: Status 404 returned error can't find the container with id ca910092d78153e8d9b5e5be2b04f0651649e2a2623725cf20817cb897817d5e Jun 10 11:06:41.466020 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.465982 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerStarted","Data":"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809"} Jun 10 11:06:41.466194 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.466031 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerStarted","Data":"ca910092d78153e8d9b5e5be2b04f0651649e2a2623725cf20817cb897817d5e"} Jun 10 11:06:41.469235 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.469211 2545 generic.go:356] "Generic (PLEG): container finished" podID="1f152be3-1b81-4121-8213-093d22a25d99" containerID="06ab8c77a04542206af82ebd0147b8af323059af28aa00e8a45656a9b2c37c7e" exitCode=0 Jun 10 11:06:41.469235 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.469233 2545 generic.go:356] "Generic (PLEG): container finished" podID="1f152be3-1b81-4121-8213-093d22a25d99" containerID="97a48d9b3ad4e1a8773b482b3c806ac152c3cad8e0ed7072a59173ee50f2fabd" exitCode=0 Jun 10 11:06:41.469463 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.469257 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerDied","Data":"06ab8c77a04542206af82ebd0147b8af323059af28aa00e8a45656a9b2c37c7e"} Jun 10 11:06:41.469463 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:41.469282 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerDied","Data":"97a48d9b3ad4e1a8773b482b3c806ac152c3cad8e0ed7072a59173ee50f2fabd"} Jun 10 11:06:46.492300 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:46.492208 2545 generic.go:356] "Generic (PLEG): container finished" podID="bbc92800-db0b-4153-b564-bdc84f06a209" containerID="c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809" exitCode=0 Jun 10 11:06:46.492300 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:46.492280 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerDied","Data":"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809"} Jun 10 11:06:47.498716 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:47.498677 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerStarted","Data":"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b"} Jun 10 11:06:49.224176 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:49.224112 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jun 10 11:06:49.224176 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:49.224165 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jun 10 11:06:52.521998 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:52.521888 2545 generic.go:356] "Generic (PLEG): container finished" podID="bbc92800-db0b-4153-b564-bdc84f06a209" containerID="eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b" exitCode=0 Jun 10 11:06:52.521998 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:52.521976 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerDied","Data":"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b"} Jun 10 11:06:53.533120 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:53.533073 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerStarted","Data":"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408"} Jun 10 11:06:53.533626 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:53.533138 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerStarted","Data":"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381"} Jun 10 11:06:53.533626 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:53.533158 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerStarted","Data":"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22"} Jun 10 11:06:53.533626 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:53.533175 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerStarted","Data":"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e"} Jun 10 11:06:53.565560 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:53.565472 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podStartSLOduration=13.565448227 podStartE2EDuration="13.565448227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:06:53.56306409 +0000 UTC m=+1315.211996633" watchObservedRunningTime="2026-06-10 11:06:53.565448227 +0000 UTC m=+1315.214380763" Jun 10 11:06:56.128606 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.128564 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:56.129131 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.128644 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:56.129294 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.129235 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:56.129570 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.129532 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:56.130467 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.130418 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Jun 10 11:06:56.130696 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.130442 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jun 10 11:06:56.130794 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.130442 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused" Jun 10 11:06:56.130794 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:56.130480 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jun 10 11:06:59.129298 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.129264 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.129777 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.129608 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.129777 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.129726 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.130165 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.130144 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.130323 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.130303 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.130431 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.130331 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.130431 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.130393 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.130558 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.130537 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.130618 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.130608 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.134303 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.134278 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.224229 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.224175 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jun 10 11:06:59.224229 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.224204 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jun 10 11:06:59.564833 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.564748 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:06:59.567684 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:06:59.567658 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:07:09.224642 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:09.224592 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.20:8443: connect: connection refused" Jun 10 11:07:09.225065 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:09.224635 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" probeResult="failure" output="Get \"https://10.134.0.20:6567/health\": dial tcp 10.134.0.20:6567: connect: connection refused" Jun 10 11:07:09.225065 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:09.224764 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:07:09.225065 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:09.224831 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:07:10.610274 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:10.610241 2545 generic.go:356] "Generic (PLEG): container finished" podID="1f152be3-1b81-4121-8213-093d22a25d99" containerID="270a4d12870bb37fbbb5c3b6b0ec40fc5239e68810a073f67053e06191071661" exitCode=137 Jun 10 11:07:10.610274 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:10.610267 2545 generic.go:356] "Generic (PLEG): container finished" podID="1f152be3-1b81-4121-8213-093d22a25d99" containerID="d7eccb19b155785a515e23a210af4288b98439ba2053813dffa194cd07fd6cf6" exitCode=137 Jun 10 11:07:10.610663 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:10.610306 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerDied","Data":"270a4d12870bb37fbbb5c3b6b0ec40fc5239e68810a073f67053e06191071661"} Jun 10 11:07:10.610663 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:10.610351 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerDied","Data":"d7eccb19b155785a515e23a210af4288b98439ba2053813dffa194cd07fd6cf6"} Jun 10 11:07:11.130648 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.130625 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:07:11.219080 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.219046 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f152be3-1b81-4121-8213-093d22a25d99-feast-data\") pod \"1f152be3-1b81-4121-8213-093d22a25d99\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " Jun 10 11:07:11.219080 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.219088 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-registry-tls\") pod \"1f152be3-1b81-4121-8213-093d22a25d99\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " Jun 10 11:07:11.219335 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.219120 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-online-tls\") pod \"1f152be3-1b81-4121-8213-093d22a25d99\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " Jun 10 11:07:11.219335 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.219140 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl64p\" (UniqueName: \"kubernetes.io/projected/1f152be3-1b81-4121-8213-093d22a25d99-kube-api-access-wl64p\") pod \"1f152be3-1b81-4121-8213-093d22a25d99\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " Jun 10 11:07:11.219335 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.219166 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-offline-tls\") pod \"1f152be3-1b81-4121-8213-093d22a25d99\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " Jun 10 11:07:11.219335 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.219237 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-ui-tls\") pod \"1f152be3-1b81-4121-8213-093d22a25d99\" (UID: \"1f152be3-1b81-4121-8213-093d22a25d99\") " Jun 10 11:07:11.219647 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.219615 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f152be3-1b81-4121-8213-093d22a25d99-feast-data" (OuterVolumeSpecName: "feast-data") pod "1f152be3-1b81-4121-8213-093d22a25d99" (UID: "1f152be3-1b81-4121-8213-093d22a25d99"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:07:11.221515 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.221484 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1f152be3-1b81-4121-8213-093d22a25d99" (UID: "1f152be3-1b81-4121-8213-093d22a25d99"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:07:11.221627 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.221511 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "1f152be3-1b81-4121-8213-093d22a25d99" (UID: "1f152be3-1b81-4121-8213-093d22a25d99"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:07:11.221627 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.221539 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-online-tls" (OuterVolumeSpecName: "online-tls") pod "1f152be3-1b81-4121-8213-093d22a25d99" (UID: "1f152be3-1b81-4121-8213-093d22a25d99"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:07:11.221726 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.221665 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f152be3-1b81-4121-8213-093d22a25d99-kube-api-access-wl64p" (OuterVolumeSpecName: "kube-api-access-wl64p") pod "1f152be3-1b81-4121-8213-093d22a25d99" (UID: "1f152be3-1b81-4121-8213-093d22a25d99"). InnerVolumeSpecName "kube-api-access-wl64p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:07:11.221726 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.221672 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "1f152be3-1b81-4121-8213-093d22a25d99" (UID: "1f152be3-1b81-4121-8213-093d22a25d99"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:07:11.320034 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.319889 2545 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-ui-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:07:11.320034 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.319936 2545 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/1f152be3-1b81-4121-8213-093d22a25d99-feast-data\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:07:11.320034 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.319948 2545 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-registry-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:07:11.320034 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.319957 2545 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-online-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:07:11.320034 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.319993 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl64p\" (UniqueName: \"kubernetes.io/projected/1f152be3-1b81-4121-8213-093d22a25d99-kube-api-access-wl64p\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:07:11.320034 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.320002 2545 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/1f152be3-1b81-4121-8213-093d22a25d99-offline-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:07:11.616861 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.616826 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" event={"ID":"1f152be3-1b81-4121-8213-093d22a25d99","Type":"ContainerDied","Data":"c645e5d972696722c881b87cedce37cd4da76e6b670c2b6903981167cccc5a66"} Jun 10 11:07:11.617409 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.616893 2545 scope.go:117] "RemoveContainer" containerID="06ab8c77a04542206af82ebd0147b8af323059af28aa00e8a45656a9b2c37c7e" Jun 10 11:07:11.617409 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.616910 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh" Jun 10 11:07:11.627080 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.627055 2545 scope.go:117] "RemoveContainer" containerID="270a4d12870bb37fbbb5c3b6b0ec40fc5239e68810a073f67053e06191071661" Jun 10 11:07:11.635283 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.635263 2545 scope.go:117] "RemoveContainer" containerID="97a48d9b3ad4e1a8773b482b3c806ac152c3cad8e0ed7072a59173ee50f2fabd" Jun 10 11:07:11.643584 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.643555 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh"] Jun 10 11:07:11.643848 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.643831 2545 scope.go:117] "RemoveContainer" containerID="d7eccb19b155785a515e23a210af4288b98439ba2053813dffa194cd07fd6cf6" Jun 10 11:07:11.649468 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.649440 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-gchnh"] Jun 10 11:07:11.652745 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.652725 2545 scope.go:117] "RemoveContainer" containerID="c2fb43952178408d1e71467d18a3eb67da71377e0305d7d4dcdc69db5c83620e" Jun 10 11:07:11.660517 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:11.660495 2545 scope.go:117] "RemoveContainer" containerID="d3545e90fcf9710099f2aa4ee41bff2dc6f2e8371cf7a9b532c5c81abf227678" Jun 10 11:07:12.782287 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:07:12.782252 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f152be3-1b81-4121-8213-093d22a25d99" path="/var/lib/kubelet/pods/1f152be3-1b81-4121-8213-093d22a25d99/volumes" Jun 10 11:08:02.701728 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.701687 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k"] Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702115 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="feast-apply" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702134 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="feast-apply" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702155 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702164 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702177 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="offline" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702186 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="offline" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702205 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="registry" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702213 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="registry" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702224 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="feast-init" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702232 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="feast-init" Jun 10 11:08:02.702245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702242 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" Jun 10 11:08:02.702787 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702252 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" Jun 10 11:08:02.702787 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702319 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="offline" Jun 10 11:08:02.702787 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702334 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="ui" Jun 10 11:08:02.702787 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702345 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="registry" Jun 10 11:08:02.702787 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.702356 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f152be3-1b81-4121-8213-093d22a25d99" containerName="online" Jun 10 11:08:02.707034 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.707010 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.711418 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.710036 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 10 11:08:02.711418 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.710548 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 10 11:08:02.711418 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.710721 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 10 11:08:02.711418 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.710873 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 10 11:08:02.711418 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.711041 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-csj7w\"" Jun 10 11:08:02.711418 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.711246 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 10 11:08:02.711418 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.711334 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 10 11:08:02.719907 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.719880 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k"] Jun 10 11:08:02.840588 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.840547 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b03c4c07-39e9-439c-83d6-25f9291fd2d5-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.840588 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.840594 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b03c4c07-39e9-439c-83d6-25f9291fd2d5-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.840809 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.840664 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lns\" (UniqueName: \"kubernetes.io/projected/b03c4c07-39e9-439c-83d6-25f9291fd2d5-kube-api-access-c5lns\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.840809 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.840745 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.840809 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.840780 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.840809 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.840808 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.941313 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.941273 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.941537 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.941319 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.941537 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.941366 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.941537 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.941458 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b03c4c07-39e9-439c-83d6-25f9291fd2d5-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.941723 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.941684 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b03c4c07-39e9-439c-83d6-25f9291fd2d5-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.941782 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.941739 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lns\" (UniqueName: \"kubernetes.io/projected/b03c4c07-39e9-439c-83d6-25f9291fd2d5-kube-api-access-c5lns\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.942017 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.941990 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b03c4c07-39e9-439c-83d6-25f9291fd2d5-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.942312 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.942294 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b03c4c07-39e9-439c-83d6-25f9291fd2d5-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.943980 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.943940 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.944078 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.943986 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.944145 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.944124 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:02.959934 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:02.959864 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lns\" (UniqueName: \"kubernetes.io/projected/b03c4c07-39e9-439c-83d6-25f9291fd2d5-kube-api-access-c5lns\") pod \"feast-simple-feast-remote-setup-669bcf45fc-t547k\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:03.019631 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:03.019591 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:03.162698 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:03.162670 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k"] Jun 10 11:08:03.164777 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:08:03.164750 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb03c4c07_39e9_439c_83d6_25f9291fd2d5.slice/crio-09cda39b9e1ac463c15edf2f682172316ec90426aa54eff6050554860ab8de01 WatchSource:0}: Error finding container 09cda39b9e1ac463c15edf2f682172316ec90426aa54eff6050554860ab8de01: Status 404 returned error can't find the container with id 09cda39b9e1ac463c15edf2f682172316ec90426aa54eff6050554860ab8de01 Jun 10 11:08:03.812101 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:03.812063 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerStarted","Data":"2cb7e110a6111cac3fc1e9a9c912088202825eacd4c606279777e785df06215d"} Jun 10 11:08:03.812485 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:03.812117 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerStarted","Data":"09cda39b9e1ac463c15edf2f682172316ec90426aa54eff6050554860ab8de01"} Jun 10 11:08:08.834345 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:08.834307 2545 generic.go:356] "Generic (PLEG): container finished" podID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerID="2cb7e110a6111cac3fc1e9a9c912088202825eacd4c606279777e785df06215d" exitCode=0 Jun 10 11:08:08.834747 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:08.834364 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerDied","Data":"2cb7e110a6111cac3fc1e9a9c912088202825eacd4c606279777e785df06215d"} Jun 10 11:08:09.839337 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:09.839304 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerStarted","Data":"ebe652b96d780fb2d8914ba0fac3fa1a232a22b8a3f74645062eefb4c92de7f2"} Jun 10 11:08:14.861627 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:14.861597 2545 generic.go:356] "Generic (PLEG): container finished" podID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerID="ebe652b96d780fb2d8914ba0fac3fa1a232a22b8a3f74645062eefb4c92de7f2" exitCode=0 Jun 10 11:08:14.862064 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:14.861633 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerDied","Data":"ebe652b96d780fb2d8914ba0fac3fa1a232a22b8a3f74645062eefb4c92de7f2"} Jun 10 11:08:15.870927 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:15.870876 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerStarted","Data":"97c416cfa0a18fe206b44bc39d3a6c8b396adf91424048c79d42a3cdc27ceb02"} Jun 10 11:08:15.871359 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:15.870947 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerStarted","Data":"d5e546d4613ad7e7954d7adf58461650f1ac920770879fe42f8982282f85fe8e"} Jun 10 11:08:15.871359 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:15.870989 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerStarted","Data":"1746a459411589c5d757f619506acc03be3a881b2e8374840e42db575d5bc883"} Jun 10 11:08:15.903856 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:15.903775 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podStartSLOduration=13.903753979 podStartE2EDuration="13.903753979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:08:15.902384424 +0000 UTC m=+1397.551316968" watchObservedRunningTime="2026-06-10 11:08:15.903753979 +0000 UTC m=+1397.552686522" Jun 10 11:08:18.020265 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:18.020223 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:18.020651 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:18.020294 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:18.020651 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:18.020314 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:18.022479 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:18.022446 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Jun 10 11:08:18.022479 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:18.022465 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jun 10 11:08:18.022618 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:18.022487 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jun 10 11:08:21.021295 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.021261 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.021806 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.021788 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.021869 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.021823 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.022093 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.022074 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.025409 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.025393 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.901473 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.901443 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.901666 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.901491 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.901891 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.901873 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:08:21.904488 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:08:21.904468 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:09:24.126245 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.126207 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k"] Jun 10 11:09:24.126842 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.126486 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" containerID="cri-o://1746a459411589c5d757f619506acc03be3a881b2e8374840e42db575d5bc883" gracePeriod=30 Jun 10 11:09:24.126842 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.126540 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" containerID="cri-o://97c416cfa0a18fe206b44bc39d3a6c8b396adf91424048c79d42a3cdc27ceb02" gracePeriod=30 Jun 10 11:09:24.126842 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.126540 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="offline" containerID="cri-o://d5e546d4613ad7e7954d7adf58461650f1ac920770879fe42f8982282f85fe8e" gracePeriod=30 Jun 10 11:09:24.283332 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.283294 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8"] Jun 10 11:09:24.283810 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.283595 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="registry" containerID="cri-o://9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e" gracePeriod=30 Jun 10 11:09:24.283810 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.283642 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="offline" containerID="cri-o://3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381" gracePeriod=30 Jun 10 11:09:24.283810 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.283611 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" containerID="cri-o://6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408" gracePeriod=30 Jun 10 11:09:24.283810 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:24.283718 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" containerID="cri-o://ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22" gracePeriod=30 Jun 10 11:09:25.152732 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.152702 2545 generic.go:356] "Generic (PLEG): container finished" podID="bbc92800-db0b-4153-b564-bdc84f06a209" containerID="6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408" exitCode=0 Jun 10 11:09:25.152732 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.152725 2545 generic.go:356] "Generic (PLEG): container finished" podID="bbc92800-db0b-4153-b564-bdc84f06a209" containerID="ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22" exitCode=0 Jun 10 11:09:25.153199 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.152777 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerDied","Data":"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408"} Jun 10 11:09:25.153199 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.152832 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerDied","Data":"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22"} Jun 10 11:09:25.155291 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.155273 2545 generic.go:356] "Generic (PLEG): container finished" podID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerID="97c416cfa0a18fe206b44bc39d3a6c8b396adf91424048c79d42a3cdc27ceb02" exitCode=0 Jun 10 11:09:25.155291 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.155288 2545 generic.go:356] "Generic (PLEG): container finished" podID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerID="1746a459411589c5d757f619506acc03be3a881b2e8374840e42db575d5bc883" exitCode=0 Jun 10 11:09:25.155429 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.155326 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerDied","Data":"97c416cfa0a18fe206b44bc39d3a6c8b396adf91424048c79d42a3cdc27ceb02"} Jun 10 11:09:25.155429 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:25.155353 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerDied","Data":"1746a459411589c5d757f619506acc03be3a881b2e8374840e42db575d5bc883"} Jun 10 11:09:29.130783 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:29.130736 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jun 10 11:09:29.565866 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:29.565778 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jun 10 11:09:31.021728 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:31.021689 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jun 10 11:09:31.901881 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:31.901837 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jun 10 11:09:39.130620 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:39.130575 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jun 10 11:09:39.565766 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:39.565685 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jun 10 11:09:41.022399 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:41.022357 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jun 10 11:09:41.901874 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:41.901833 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jun 10 11:09:49.130830 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:49.130775 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Jun 10 11:09:49.131279 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:49.130921 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:09:49.565353 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:49.565251 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Jun 10 11:09:49.565528 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:49.565443 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:09:51.022396 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:51.022355 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Jun 10 11:09:51.022829 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:51.022461 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:09:51.902304 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:51.902268 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Jun 10 11:09:51.902499 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:51.902399 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:09:54.264937 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.264909 2545 generic.go:356] "Generic (PLEG): container finished" podID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerID="d5e546d4613ad7e7954d7adf58461650f1ac920770879fe42f8982282f85fe8e" exitCode=137 Jun 10 11:09:54.265279 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.264981 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerDied","Data":"d5e546d4613ad7e7954d7adf58461650f1ac920770879fe42f8982282f85fe8e"} Jun 10 11:09:54.322119 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.322095 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:09:54.418612 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.418547 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-offline-tls\") pod \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " Jun 10 11:09:54.418612 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.418577 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-online-tls\") pod \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " Jun 10 11:09:54.418612 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.418595 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-ui-tls\") pod \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " Jun 10 11:09:54.418833 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.418637 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b03c4c07-39e9-439c-83d6-25f9291fd2d5-registry-tls\") pod \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " Jun 10 11:09:54.418833 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.418668 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5lns\" (UniqueName: \"kubernetes.io/projected/b03c4c07-39e9-439c-83d6-25f9291fd2d5-kube-api-access-c5lns\") pod \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " Jun 10 11:09:54.418833 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.418721 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b03c4c07-39e9-439c-83d6-25f9291fd2d5-feast-data\") pod \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\" (UID: \"b03c4c07-39e9-439c-83d6-25f9291fd2d5\") " Jun 10 11:09:54.419092 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.419065 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03c4c07-39e9-439c-83d6-25f9291fd2d5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b03c4c07-39e9-439c-83d6-25f9291fd2d5" (UID: "b03c4c07-39e9-439c-83d6-25f9291fd2d5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 11:09:54.419321 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.419266 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03c4c07-39e9-439c-83d6-25f9291fd2d5-feast-data" (OuterVolumeSpecName: "feast-data") pod "b03c4c07-39e9-439c-83d6-25f9291fd2d5" (UID: "b03c4c07-39e9-439c-83d6-25f9291fd2d5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:09:54.420761 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.420740 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b03c4c07-39e9-439c-83d6-25f9291fd2d5" (UID: "b03c4c07-39e9-439c-83d6-25f9291fd2d5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:09:54.420840 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.420775 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-online-tls" (OuterVolumeSpecName: "online-tls") pod "b03c4c07-39e9-439c-83d6-25f9291fd2d5" (UID: "b03c4c07-39e9-439c-83d6-25f9291fd2d5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:09:54.420840 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.420787 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03c4c07-39e9-439c-83d6-25f9291fd2d5-kube-api-access-c5lns" (OuterVolumeSpecName: "kube-api-access-c5lns") pod "b03c4c07-39e9-439c-83d6-25f9291fd2d5" (UID: "b03c4c07-39e9-439c-83d6-25f9291fd2d5"). InnerVolumeSpecName "kube-api-access-c5lns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:09:54.420840 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.420812 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b03c4c07-39e9-439c-83d6-25f9291fd2d5" (UID: "b03c4c07-39e9-439c-83d6-25f9291fd2d5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:09:54.520104 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.520080 2545 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/b03c4c07-39e9-439c-83d6-25f9291fd2d5-registry-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:54.520104 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.520104 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lns\" (UniqueName: \"kubernetes.io/projected/b03c4c07-39e9-439c-83d6-25f9291fd2d5-kube-api-access-c5lns\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:54.520236 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.520114 2545 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b03c4c07-39e9-439c-83d6-25f9291fd2d5-feast-data\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:54.520236 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.520122 2545 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-offline-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:54.520236 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.520131 2545 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-online-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:54.520236 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.520138 2545 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b03c4c07-39e9-439c-83d6-25f9291fd2d5-ui-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:54.923166 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:54.923138 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:09:55.024382 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.024353 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-offline-tls\") pod \"bbc92800-db0b-4153-b564-bdc84f06a209\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " Jun 10 11:09:55.024515 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.024405 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pvz9\" (UniqueName: \"kubernetes.io/projected/bbc92800-db0b-4153-b564-bdc84f06a209-kube-api-access-8pvz9\") pod \"bbc92800-db0b-4153-b564-bdc84f06a209\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " Jun 10 11:09:55.024515 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.024433 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-ui-tls\") pod \"bbc92800-db0b-4153-b564-bdc84f06a209\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " Jun 10 11:09:55.024515 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.024448 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-online-tls\") pod \"bbc92800-db0b-4153-b564-bdc84f06a209\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " Jun 10 11:09:55.024515 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.024480 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc92800-db0b-4153-b564-bdc84f06a209-feast-data\") pod \"bbc92800-db0b-4153-b564-bdc84f06a209\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " Jun 10 11:09:55.024515 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.024499 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-registry-tls\") pod \"bbc92800-db0b-4153-b564-bdc84f06a209\" (UID: \"bbc92800-db0b-4153-b564-bdc84f06a209\") " Jun 10 11:09:55.025062 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.025029 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc92800-db0b-4153-b564-bdc84f06a209-feast-data" (OuterVolumeSpecName: "feast-data") pod "bbc92800-db0b-4153-b564-bdc84f06a209" (UID: "bbc92800-db0b-4153-b564-bdc84f06a209"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:09:55.026501 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.026470 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-online-tls" (OuterVolumeSpecName: "online-tls") pod "bbc92800-db0b-4153-b564-bdc84f06a209" (UID: "bbc92800-db0b-4153-b564-bdc84f06a209"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:09:55.026598 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.026514 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bbc92800-db0b-4153-b564-bdc84f06a209" (UID: "bbc92800-db0b-4153-b564-bdc84f06a209"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:09:55.026598 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.026553 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bbc92800-db0b-4153-b564-bdc84f06a209" (UID: "bbc92800-db0b-4153-b564-bdc84f06a209"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:09:55.026839 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.026821 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bbc92800-db0b-4153-b564-bdc84f06a209" (UID: "bbc92800-db0b-4153-b564-bdc84f06a209"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:09:55.026925 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.026843 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc92800-db0b-4153-b564-bdc84f06a209-kube-api-access-8pvz9" (OuterVolumeSpecName: "kube-api-access-8pvz9") pod "bbc92800-db0b-4153-b564-bdc84f06a209" (UID: "bbc92800-db0b-4153-b564-bdc84f06a209"). InnerVolumeSpecName "kube-api-access-8pvz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:09:55.125355 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.125330 2545 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-ui-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:55.125355 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.125355 2545 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-online-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:55.125508 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.125365 2545 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc92800-db0b-4153-b564-bdc84f06a209-feast-data\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:55.125508 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.125373 2545 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-registry-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:55.125508 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.125387 2545 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bbc92800-db0b-4153-b564-bdc84f06a209-offline-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:55.125508 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.125396 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pvz9\" (UniqueName: \"kubernetes.io/projected/bbc92800-db0b-4153-b564-bdc84f06a209-kube-api-access-8pvz9\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:09:55.271756 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.271718 2545 generic.go:356] "Generic (PLEG): container finished" podID="bbc92800-db0b-4153-b564-bdc84f06a209" containerID="3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381" exitCode=137 Jun 10 11:09:55.271756 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.271747 2545 generic.go:356] "Generic (PLEG): container finished" podID="bbc92800-db0b-4153-b564-bdc84f06a209" containerID="9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e" exitCode=137 Jun 10 11:09:55.272256 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.271808 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" Jun 10 11:09:55.272256 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.271796 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerDied","Data":"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381"} Jun 10 11:09:55.272256 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.271920 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerDied","Data":"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e"} Jun 10 11:09:55.272256 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.271946 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8" event={"ID":"bbc92800-db0b-4153-b564-bdc84f06a209","Type":"ContainerDied","Data":"ca910092d78153e8d9b5e5be2b04f0651649e2a2623725cf20817cb897817d5e"} Jun 10 11:09:55.272256 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.271987 2545 scope.go:117] "RemoveContainer" containerID="6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408" Jun 10 11:09:55.274620 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.274603 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" Jun 10 11:09:55.274734 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.274604 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k" event={"ID":"b03c4c07-39e9-439c-83d6-25f9291fd2d5","Type":"ContainerDied","Data":"09cda39b9e1ac463c15edf2f682172316ec90426aa54eff6050554860ab8de01"} Jun 10 11:09:55.281737 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.281723 2545 scope.go:117] "RemoveContainer" containerID="3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381" Jun 10 11:09:55.288651 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.288634 2545 scope.go:117] "RemoveContainer" containerID="ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22" Jun 10 11:09:55.294558 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.294535 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k"] Jun 10 11:09:55.296827 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.296808 2545 scope.go:117] "RemoveContainer" containerID="9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e" Jun 10 11:09:55.298445 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.298427 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-t547k"] Jun 10 11:09:55.303641 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.303625 2545 scope.go:117] "RemoveContainer" containerID="eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b" Jun 10 11:09:55.311065 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.311025 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8"] Jun 10 11:09:55.312038 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.311227 2545 scope.go:117] "RemoveContainer" containerID="c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809" Jun 10 11:09:55.314955 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.314936 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-t5kf8"] Jun 10 11:09:55.319007 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.318982 2545 scope.go:117] "RemoveContainer" containerID="6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408" Jun 10 11:09:55.319245 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:09:55.319229 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408\": container with ID starting with 6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408 not found: ID does not exist" containerID="6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408" Jun 10 11:09:55.319300 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319250 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408"} err="failed to get container status \"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408\": rpc error: code = NotFound desc = could not find container \"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408\": container with ID starting with 6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408 not found: ID does not exist" Jun 10 11:09:55.319300 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319267 2545 scope.go:117] "RemoveContainer" containerID="3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381" Jun 10 11:09:55.319470 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:09:55.319452 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381\": container with ID starting with 3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381 not found: ID does not exist" containerID="3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381" Jun 10 11:09:55.319529 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319480 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381"} err="failed to get container status \"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381\": rpc error: code = NotFound desc = could not find container \"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381\": container with ID starting with 3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381 not found: ID does not exist" Jun 10 11:09:55.319529 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319504 2545 scope.go:117] "RemoveContainer" containerID="ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22" Jun 10 11:09:55.319723 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:09:55.319707 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22\": container with ID starting with ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22 not found: ID does not exist" containerID="ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22" Jun 10 11:09:55.319757 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319728 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22"} err="failed to get container status \"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22\": rpc error: code = NotFound desc = could not find container \"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22\": container with ID starting with ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22 not found: ID does not exist" Jun 10 11:09:55.319757 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319742 2545 scope.go:117] "RemoveContainer" containerID="9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e" Jun 10 11:09:55.319929 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:09:55.319914 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e\": container with ID starting with 9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e not found: ID does not exist" containerID="9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e" Jun 10 11:09:55.319996 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319930 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e"} err="failed to get container status \"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e\": rpc error: code = NotFound desc = could not find container \"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e\": container with ID starting with 9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e not found: ID does not exist" Jun 10 11:09:55.319996 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.319942 2545 scope.go:117] "RemoveContainer" containerID="eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b" Jun 10 11:09:55.320178 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:09:55.320164 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b\": container with ID starting with eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b not found: ID does not exist" containerID="eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b" Jun 10 11:09:55.320219 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320181 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b"} err="failed to get container status \"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b\": rpc error: code = NotFound desc = could not find container \"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b\": container with ID starting with eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b not found: ID does not exist" Jun 10 11:09:55.320219 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320193 2545 scope.go:117] "RemoveContainer" containerID="c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809" Jun 10 11:09:55.320382 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:09:55.320367 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809\": container with ID starting with c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809 not found: ID does not exist" containerID="c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809" Jun 10 11:09:55.320425 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320384 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809"} err="failed to get container status \"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809\": rpc error: code = NotFound desc = could not find container \"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809\": container with ID starting with c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809 not found: ID does not exist" Jun 10 11:09:55.320425 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320396 2545 scope.go:117] "RemoveContainer" containerID="6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408" Jun 10 11:09:55.320570 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320555 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408"} err="failed to get container status \"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408\": rpc error: code = NotFound desc = could not find container \"6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408\": container with ID starting with 6cbb5547611b0c6a7920c01a5aece08dce66e67e1859274ad921496355fca408 not found: ID does not exist" Jun 10 11:09:55.320570 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320569 2545 scope.go:117] "RemoveContainer" containerID="3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381" Jun 10 11:09:55.320742 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320727 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381"} err="failed to get container status \"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381\": rpc error: code = NotFound desc = could not find container \"3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381\": container with ID starting with 3ebd05b4004b421790295507d1fe2a3f92cb0c976ce4884afa707d80fd1df381 not found: ID does not exist" Jun 10 11:09:55.320742 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320741 2545 scope.go:117] "RemoveContainer" containerID="ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22" Jun 10 11:09:55.320933 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320916 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22"} err="failed to get container status \"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22\": rpc error: code = NotFound desc = could not find container \"ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22\": container with ID starting with ab19a43423914a1292342a6c560f085b5931ba32cd18ecceaa8255611ef98b22 not found: ID does not exist" Jun 10 11:09:55.320933 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.320933 2545 scope.go:117] "RemoveContainer" containerID="9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e" Jun 10 11:09:55.321142 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.321126 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e"} err="failed to get container status \"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e\": rpc error: code = NotFound desc = could not find container \"9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e\": container with ID starting with 9f80a84b49a69830670fa9f324cc641b8b3200a89e852759af7f0924e329f82e not found: ID does not exist" Jun 10 11:09:55.321142 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.321141 2545 scope.go:117] "RemoveContainer" containerID="eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b" Jun 10 11:09:55.321332 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.321313 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b"} err="failed to get container status \"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b\": rpc error: code = NotFound desc = could not find container \"eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b\": container with ID starting with eeb13c8801bff12f5111c18179b9ecdbee7bb0aed1ac5450d6dc2ee326169b7b not found: ID does not exist" Jun 10 11:09:55.321411 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.321333 2545 scope.go:117] "RemoveContainer" containerID="c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809" Jun 10 11:09:55.321530 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.321515 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809"} err="failed to get container status \"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809\": rpc error: code = NotFound desc = could not find container \"c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809\": container with ID starting with c643b67091ca6ea519f7c9bbdb439af1c32679c47bc5e6b5e1a2a051d075f809 not found: ID does not exist" Jun 10 11:09:55.321530 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.321529 2545 scope.go:117] "RemoveContainer" containerID="97c416cfa0a18fe206b44bc39d3a6c8b396adf91424048c79d42a3cdc27ceb02" Jun 10 11:09:55.328359 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.328345 2545 scope.go:117] "RemoveContainer" containerID="d5e546d4613ad7e7954d7adf58461650f1ac920770879fe42f8982282f85fe8e" Jun 10 11:09:55.335022 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.335005 2545 scope.go:117] "RemoveContainer" containerID="1746a459411589c5d757f619506acc03be3a881b2e8374840e42db575d5bc883" Jun 10 11:09:55.341507 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.341494 2545 scope.go:117] "RemoveContainer" containerID="ebe652b96d780fb2d8914ba0fac3fa1a232a22b8a3f74645062eefb4c92de7f2" Jun 10 11:09:55.347940 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:55.347925 2545 scope.go:117] "RemoveContainer" containerID="2cb7e110a6111cac3fc1e9a9c912088202825eacd4c606279777e785df06215d" Jun 10 11:09:56.781404 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:56.781366 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" path="/var/lib/kubelet/pods/b03c4c07-39e9-439c-83d6-25f9291fd2d5/volumes" Jun 10 11:09:56.781888 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:09:56.781875 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" path="/var/lib/kubelet/pods/bbc92800-db0b-4153-b564-bdc84f06a209/volumes" Jun 10 11:10:13.322235 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:13.322141 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2"] Jun 10 11:10:13.322685 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:13.322396 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" podUID="2617864f-c265-4f54-8003-d3d75b23d2d3" containerName="manager" containerID="cri-o://5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0" gracePeriod=10 Jun 10 11:10:13.549622 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:13.549599 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:10:13.569422 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:13.569392 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5dsl\" (UniqueName: \"kubernetes.io/projected/2617864f-c265-4f54-8003-d3d75b23d2d3-kube-api-access-t5dsl\") pod \"2617864f-c265-4f54-8003-d3d75b23d2d3\" (UID: \"2617864f-c265-4f54-8003-d3d75b23d2d3\") " Jun 10 11:10:13.571798 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:13.571765 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2617864f-c265-4f54-8003-d3d75b23d2d3-kube-api-access-t5dsl" (OuterVolumeSpecName: "kube-api-access-t5dsl") pod "2617864f-c265-4f54-8003-d3d75b23d2d3" (UID: "2617864f-c265-4f54-8003-d3d75b23d2d3"). InnerVolumeSpecName "kube-api-access-t5dsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:10:13.670791 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:13.670761 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5dsl\" (UniqueName: \"kubernetes.io/projected/2617864f-c265-4f54-8003-d3d75b23d2d3-kube-api-access-t5dsl\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:10:14.350466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.350430 2545 generic.go:356] "Generic (PLEG): container finished" podID="2617864f-c265-4f54-8003-d3d75b23d2d3" containerID="5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0" exitCode=0 Jun 10 11:10:14.350839 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.350471 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" Jun 10 11:10:14.350839 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.350505 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" event={"ID":"2617864f-c265-4f54-8003-d3d75b23d2d3","Type":"ContainerDied","Data":"5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0"} Jun 10 11:10:14.350839 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.350562 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2" event={"ID":"2617864f-c265-4f54-8003-d3d75b23d2d3","Type":"ContainerDied","Data":"f66f40c2f319c16d0aa160eef2af7168076ff03d39ad82430441dffe955a5dbe"} Jun 10 11:10:14.350839 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.350585 2545 scope.go:117] "RemoveContainer" containerID="5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0" Jun 10 11:10:14.358804 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.358788 2545 scope.go:117] "RemoveContainer" containerID="5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0" Jun 10 11:10:14.359052 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:10:14.359030 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0\": container with ID starting with 5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0 not found: ID does not exist" containerID="5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0" Jun 10 11:10:14.359107 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.359059 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0"} err="failed to get container status \"5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0\": rpc error: code = NotFound desc = could not find container \"5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0\": container with ID starting with 5e0d866f065a3e89fdd2a20ca9bc82d523c9b935416c911289856233b066e1b0 not found: ID does not exist" Jun 10 11:10:14.381647 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.381622 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2"] Jun 10 11:10:14.386562 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.386544 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-lktj2"] Jun 10 11:10:14.781362 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:14.781337 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2617864f-c265-4f54-8003-d3d75b23d2d3" path="/var/lib/kubelet/pods/2617864f-c265-4f54-8003-d3d75b23d2d3/volumes" Jun 10 11:10:18.038436 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038405 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh"] Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038657 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038667 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038677 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038682 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038689 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038695 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038704 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038709 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038715 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="offline" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038719 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="offline" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038726 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="offline" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038731 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="offline" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038737 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="feast-apply" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038742 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="feast-apply" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038750 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="registry" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038754 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="registry" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038762 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="feast-init" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038767 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="feast-init" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038774 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2617864f-c265-4f54-8003-d3d75b23d2d3" containerName="manager" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038779 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="2617864f-c265-4f54-8003-d3d75b23d2d3" containerName="manager" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038787 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="feast-apply" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038791 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="feast-apply" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038797 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="feast-init" Jun 10 11:10:18.038796 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038801 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="feast-init" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038838 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="offline" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038844 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="ui" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038850 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="registry" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038856 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="online" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038863 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc92800-db0b-4153-b564-bdc84f06a209" containerName="ui" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038868 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="offline" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038873 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="2617864f-c265-4f54-8003-d3d75b23d2d3" containerName="manager" Jun 10 11:10:18.039466 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.038878 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03c4c07-39e9-439c-83d6-25f9291fd2d5" containerName="online" Jun 10 11:10:18.043045 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.043029 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" Jun 10 11:10:18.048773 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.048754 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jun 10 11:10:18.049979 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.049945 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-stjlw\"" Jun 10 11:10:18.050070 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.049988 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jun 10 11:10:18.061749 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.061728 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh"] Jun 10 11:10:18.101283 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.101257 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7km\" (UniqueName: \"kubernetes.io/projected/c815a564-b524-48f1-981a-276e6b55585f-kube-api-access-gf7km\") pod \"feast-operator-controller-manager-6d54c7db9d-45wjh\" (UID: \"c815a564-b524-48f1-981a-276e6b55585f\") " pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" Jun 10 11:10:18.202048 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.202021 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7km\" (UniqueName: \"kubernetes.io/projected/c815a564-b524-48f1-981a-276e6b55585f-kube-api-access-gf7km\") pod \"feast-operator-controller-manager-6d54c7db9d-45wjh\" (UID: \"c815a564-b524-48f1-981a-276e6b55585f\") " pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" Jun 10 11:10:18.213477 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.213453 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7km\" (UniqueName: \"kubernetes.io/projected/c815a564-b524-48f1-981a-276e6b55585f-kube-api-access-gf7km\") pod \"feast-operator-controller-manager-6d54c7db9d-45wjh\" (UID: \"c815a564-b524-48f1-981a-276e6b55585f\") " pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" Jun 10 11:10:18.352682 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.352596 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" Jun 10 11:10:18.472843 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:18.472814 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh"] Jun 10 11:10:18.476200 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:10:18.476176 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc815a564_b524_48f1_981a_276e6b55585f.slice/crio-217aaeb55bf7b8a68eb545b5f77d27566000d613cd419ad0df542dbe2471e55a WatchSource:0}: Error finding container 217aaeb55bf7b8a68eb545b5f77d27566000d613cd419ad0df542dbe2471e55a: Status 404 returned error can't find the container with id 217aaeb55bf7b8a68eb545b5f77d27566000d613cd419ad0df542dbe2471e55a Jun 10 11:10:19.371501 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:19.371466 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" event={"ID":"c815a564-b524-48f1-981a-276e6b55585f","Type":"ContainerStarted","Data":"9f64e49ca58722246360a35ad3fa474f18425e6d90e6c249605d2a5de0b69b33"} Jun 10 11:10:19.371865 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:19.371520 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" event={"ID":"c815a564-b524-48f1-981a-276e6b55585f","Type":"ContainerStarted","Data":"217aaeb55bf7b8a68eb545b5f77d27566000d613cd419ad0df542dbe2471e55a"} Jun 10 11:10:19.371865 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:19.371582 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" Jun 10 11:10:19.390613 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:19.390552 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" podStartSLOduration=1.390516382 podStartE2EDuration="1.390516382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:10:19.389499751 +0000 UTC m=+1521.038432292" watchObservedRunningTime="2026-06-10 11:10:19.390516382 +0000 UTC m=+1521.039448922" Jun 10 11:10:30.378109 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:30.378075 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-6d54c7db9d-45wjh" Jun 10 11:10:35.852643 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.852608 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz"] Jun 10 11:10:35.856079 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.856062 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:35.859171 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.859149 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jun 10 11:10:35.859171 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.859159 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jun 10 11:10:35.859321 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.859212 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jun 10 11:10:35.859321 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.859248 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jun 10 11:10:35.859321 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.859259 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-gqwb5\"" Jun 10 11:10:35.859321 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.859311 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jun 10 11:10:35.859535 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.859519 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jun 10 11:10:35.870904 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.870878 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz"] Jun 10 11:10:35.933698 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.933664 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:35.933698 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.933702 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17dd1591-53a4-4245-a683-76cfaf4e3f53-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:35.933888 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.933743 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:35.933888 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.933793 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:35.933888 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.933813 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbchg\" (UniqueName: \"kubernetes.io/projected/17dd1591-53a4-4245-a683-76cfaf4e3f53-kube-api-access-dbchg\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:35.933888 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:35.933856 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.035124 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.035089 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.035124 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.035126 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.035347 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.035153 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17dd1591-53a4-4245-a683-76cfaf4e3f53-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.035347 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.035180 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.035347 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.035231 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.035347 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:10:36.035240 2545 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jun 10 11:10:36.035347 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.035256 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbchg\" (UniqueName: \"kubernetes.io/projected/17dd1591-53a4-4245-a683-76cfaf4e3f53-kube-api-access-dbchg\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.035347 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:10:36.035316 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls podName:17dd1591-53a4-4245-a683-76cfaf4e3f53 nodeName:}" failed. No retries permitted until 2026-06-10 11:10:36.535294345 +0000 UTC m=+1538.184226864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls") pod "feast-simple-feast-setup-75c64c888b-6k5mz" (UID: "17dd1591-53a4-4245-a683-76cfaf4e3f53") : secret "feast-simple-feast-setup-offline-tls" not found Jun 10 11:10:36.035618 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.035558 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17dd1591-53a4-4245-a683-76cfaf4e3f53-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.037652 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.037630 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.037757 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.037672 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.037757 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.037672 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.044721 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.044699 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbchg\" (UniqueName: \"kubernetes.io/projected/17dd1591-53a4-4245-a683-76cfaf4e3f53-kube-api-access-dbchg\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.540790 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.540755 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.543175 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.543153 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-6k5mz\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.765256 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.765216 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:36.912668 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:36.912639 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz"] Jun 10 11:10:36.913930 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:10:36.913903 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17dd1591_53a4_4245_a683_76cfaf4e3f53.slice/crio-9868f2196d473506ca9a646d3d5407e42ef246bc11004be35e92c57004862116 WatchSource:0}: Error finding container 9868f2196d473506ca9a646d3d5407e42ef246bc11004be35e92c57004862116: Status 404 returned error can't find the container with id 9868f2196d473506ca9a646d3d5407e42ef246bc11004be35e92c57004862116 Jun 10 11:10:37.437524 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:37.437484 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerStarted","Data":"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e"} Jun 10 11:10:37.437524 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:37.437529 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerStarted","Data":"9868f2196d473506ca9a646d3d5407e42ef246bc11004be35e92c57004862116"} Jun 10 11:10:41.453174 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:41.453144 2545 generic.go:356] "Generic (PLEG): container finished" podID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerID="d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e" exitCode=0 Jun 10 11:10:41.453591 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:41.453196 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerDied","Data":"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e"} Jun 10 11:10:42.459336 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:42.459303 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerStarted","Data":"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3"} Jun 10 11:10:47.478919 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:47.478884 2545 generic.go:356] "Generic (PLEG): container finished" podID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerID="a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3" exitCode=0 Jun 10 11:10:47.478919 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:47.478922 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerDied","Data":"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3"} Jun 10 11:10:48.498068 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.498018 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerStarted","Data":"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521"} Jun 10 11:10:48.498542 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.498076 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerStarted","Data":"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865"} Jun 10 11:10:48.498542 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.498097 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerStarted","Data":"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f"} Jun 10 11:10:48.498542 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.498113 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerStarted","Data":"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8"} Jun 10 11:10:48.532218 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.532139 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podStartSLOduration=13.53211744 podStartE2EDuration="13.53211744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:10:48.530117584 +0000 UTC m=+1550.179050127" watchObservedRunningTime="2026-06-10 11:10:48.53211744 +0000 UTC m=+1550.181049983" Jun 10 11:10:48.766282 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.766186 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:48.766282 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.766248 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:48.766282 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.766272 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:48.766553 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.766290 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:48.768403 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.768365 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Jun 10 11:10:48.768644 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.768413 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Jun 10 11:10:48.768748 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.768413 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jun 10 11:10:48.768748 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:48.768457 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jun 10 11:10:51.766311 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:51.766268 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Jun 10 11:10:51.766867 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:51.766268 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Jun 10 11:10:51.766959 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:51.766268 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jun 10 11:10:51.766959 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:51.766429 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jun 10 11:10:54.767003 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.766947 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767367 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767092 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767367 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767184 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767367 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767217 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767367 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767272 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767367 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767310 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767640 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767612 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767678 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767657 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.767710 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.767689 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:54.771449 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:54.771434 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:55.534039 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:55.534004 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:10:55.536814 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:10:55.536795 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:11:58.086160 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.086123 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz"] Jun 10 11:11:58.086687 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.086419 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="registry" containerID="cri-o://b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8" gracePeriod=30 Jun 10 11:11:58.086687 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.086466 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" containerID="cri-o://bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f" gracePeriod=30 Jun 10 11:11:58.086687 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.086494 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" containerID="cri-o://2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521" gracePeriod=30 Jun 10 11:11:58.086687 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.086459 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="offline" containerID="cri-o://bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865" gracePeriod=30 Jun 10 11:11:58.444193 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.443672 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m"] Jun 10 11:11:58.447632 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.447603 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.454776 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.454754 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4svb5\"" Jun 10 11:11:58.486754 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.486719 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m"] Jun 10 11:11:58.613097 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.613072 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.613214 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.613104 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.613214 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.613122 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.613214 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.613157 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ab609-8eb4-405c-8168-1be8cce83065-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.613214 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.613178 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8gp\" (UniqueName: \"kubernetes.io/projected/fb0ab609-8eb4-405c-8168-1be8cce83065-kube-api-access-lz8gp\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.613214 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.613199 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.713857 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.713834 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.713937 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.713865 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.713937 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.713889 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.713937 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.713931 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ab609-8eb4-405c-8168-1be8cce83065-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.714079 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.713960 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8gp\" (UniqueName: \"kubernetes.io/projected/fb0ab609-8eb4-405c-8168-1be8cce83065-kube-api-access-lz8gp\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.714079 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.714013 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.714386 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.714354 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ab609-8eb4-405c-8168-1be8cce83065-feast-data\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.716432 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.716399 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-registry-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.716731 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.716710 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-ui-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.716804 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.716708 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-online-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.716804 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.716753 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-offline-tls\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.724901 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.724879 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8gp\" (UniqueName: \"kubernetes.io/projected/fb0ab609-8eb4-405c-8168-1be8cce83065-kube-api-access-lz8gp\") pod \"feast-simple-feast-setup-75c64c888b-q7g8m\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.762277 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.762249 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4svb5\"" Jun 10 11:11:58.765514 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.765489 2545 generic.go:356] "Generic (PLEG): container finished" podID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerID="2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521" exitCode=0 Jun 10 11:11:58.765604 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.765548 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerDied","Data":"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521"} Jun 10 11:11:58.767640 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.767624 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:11:58.906989 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:58.906946 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m"] Jun 10 11:11:58.908280 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:11:58.908255 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb0ab609_8eb4_405c_8168_1be8cce83065.slice/crio-8fdf1446f2ccb4d9edf9206dd50c9a101e3333003b4f1632ed27ccd8cc634cc7 WatchSource:0}: Error finding container 8fdf1446f2ccb4d9edf9206dd50c9a101e3333003b4f1632ed27ccd8cc634cc7: Status 404 returned error can't find the container with id 8fdf1446f2ccb4d9edf9206dd50c9a101e3333003b4f1632ed27ccd8cc634cc7 Jun 10 11:11:59.772281 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:59.772246 2545 generic.go:356] "Generic (PLEG): container finished" podID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerID="bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f" exitCode=0 Jun 10 11:11:59.772747 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:59.772314 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerDied","Data":"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f"} Jun 10 11:11:59.773704 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:59.773680 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerStarted","Data":"5e2b6b4e55c41bd6aaf5b17ce5b86b3eb9099c06a14b88842e49f8e5e0d5dc13"} Jun 10 11:11:59.773826 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:11:59.773716 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerStarted","Data":"8fdf1446f2ccb4d9edf9206dd50c9a101e3333003b4f1632ed27ccd8cc634cc7"} Jun 10 11:12:03.789066 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:03.789032 2545 generic.go:356] "Generic (PLEG): container finished" podID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerID="5e2b6b4e55c41bd6aaf5b17ce5b86b3eb9099c06a14b88842e49f8e5e0d5dc13" exitCode=0 Jun 10 11:12:03.789487 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:03.789105 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerDied","Data":"5e2b6b4e55c41bd6aaf5b17ce5b86b3eb9099c06a14b88842e49f8e5e0d5dc13"} Jun 10 11:12:04.767832 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:04.767789 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jun 10 11:12:04.794121 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:04.794092 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerStarted","Data":"841c97f1337cfb9e402a3228c71746f6f0f553e72d024b5ad2931a1b65f69e80"} Jun 10 11:12:05.534438 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:05.534398 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jun 10 11:12:09.814978 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:09.814942 2545 generic.go:356] "Generic (PLEG): container finished" podID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerID="841c97f1337cfb9e402a3228c71746f6f0f553e72d024b5ad2931a1b65f69e80" exitCode=0 Jun 10 11:12:09.815351 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:09.815014 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerDied","Data":"841c97f1337cfb9e402a3228c71746f6f0f553e72d024b5ad2931a1b65f69e80"} Jun 10 11:12:10.824762 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:10.824723 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerStarted","Data":"7ef6865ffc49d947ae1b150687f9e1196ff9c73d4c92cf7350e176678809e4a7"} Jun 10 11:12:10.825242 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:10.824784 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerStarted","Data":"92bb69e1db0515e5a7e84c02132ab56ce7f251d8ca156df62ea5e6e377030cfe"} Jun 10 11:12:10.825242 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:10.824804 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerStarted","Data":"a94b7055061f8c1d7af3da4bad56fa587fac989c11e0e891fc60f85de218ade8"} Jun 10 11:12:10.825242 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:10.824821 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerStarted","Data":"12b22499161d63bae1c9c36146cc85ddf40be81372a28c8915ad38f959cc6147"} Jun 10 11:12:10.858568 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:10.858220 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podStartSLOduration=12.858197521 podStartE2EDuration="12.858197521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:12:10.85670786 +0000 UTC m=+1632.505640425" watchObservedRunningTime="2026-06-10 11:12:10.858197521 +0000 UTC m=+1632.507130062" Jun 10 11:12:13.768366 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.768333 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:13.768976 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.768943 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:13.769103 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.769000 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:13.769103 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.769021 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:13.770676 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.770643 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Jun 10 11:12:13.770877 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.770682 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Jun 10 11:12:13.771003 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.770690 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Jun 10 11:12:13.771003 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:13.770722 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Jun 10 11:12:14.768317 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:14.768103 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jun 10 11:12:15.534400 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:15.534362 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jun 10 11:12:16.769609 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.769581 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770022 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.769705 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770022 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.770010 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770513 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.770492 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770643 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.770613 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770643 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.770634 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770764 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.770753 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770819 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.770809 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.770949 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.770934 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.773989 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.773977 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.857280 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.857249 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:16.860057 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:16.860027 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:12:24.768080 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:24.768029 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Jun 10 11:12:24.768473 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:24.768198 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:12:25.535062 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:25.535015 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Jun 10 11:12:25.535214 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:25.535145 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:12:28.737504 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.737476 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:12:28.754082 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.754060 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-online-tls\") pod \"17dd1591-53a4-4245-a683-76cfaf4e3f53\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " Jun 10 11:12:28.754218 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.754093 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls\") pod \"17dd1591-53a4-4245-a683-76cfaf4e3f53\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " Jun 10 11:12:28.754218 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.754108 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-ui-tls\") pod \"17dd1591-53a4-4245-a683-76cfaf4e3f53\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " Jun 10 11:12:28.754327 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.754235 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbchg\" (UniqueName: \"kubernetes.io/projected/17dd1591-53a4-4245-a683-76cfaf4e3f53-kube-api-access-dbchg\") pod \"17dd1591-53a4-4245-a683-76cfaf4e3f53\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " Jun 10 11:12:28.754327 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.754279 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17dd1591-53a4-4245-a683-76cfaf4e3f53-feast-data\") pod \"17dd1591-53a4-4245-a683-76cfaf4e3f53\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " Jun 10 11:12:28.754327 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.754314 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-registry-tls\") pod \"17dd1591-53a4-4245-a683-76cfaf4e3f53\" (UID: \"17dd1591-53a4-4245-a683-76cfaf4e3f53\") " Jun 10 11:12:28.754829 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.754790 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dd1591-53a4-4245-a683-76cfaf4e3f53-feast-data" (OuterVolumeSpecName: "feast-data") pod "17dd1591-53a4-4245-a683-76cfaf4e3f53" (UID: "17dd1591-53a4-4245-a683-76cfaf4e3f53"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:12:28.756272 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.756246 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-online-tls" (OuterVolumeSpecName: "online-tls") pod "17dd1591-53a4-4245-a683-76cfaf4e3f53" (UID: "17dd1591-53a4-4245-a683-76cfaf4e3f53"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:12:28.756533 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.756501 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "17dd1591-53a4-4245-a683-76cfaf4e3f53" (UID: "17dd1591-53a4-4245-a683-76cfaf4e3f53"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:12:28.756618 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.756556 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "17dd1591-53a4-4245-a683-76cfaf4e3f53" (UID: "17dd1591-53a4-4245-a683-76cfaf4e3f53"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:12:28.756618 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.756597 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "17dd1591-53a4-4245-a683-76cfaf4e3f53" (UID: "17dd1591-53a4-4245-a683-76cfaf4e3f53"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:12:28.756690 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.756651 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dd1591-53a4-4245-a683-76cfaf4e3f53-kube-api-access-dbchg" (OuterVolumeSpecName: "kube-api-access-dbchg") pod "17dd1591-53a4-4245-a683-76cfaf4e3f53" (UID: "17dd1591-53a4-4245-a683-76cfaf4e3f53"). InnerVolumeSpecName "kube-api-access-dbchg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:12:28.855771 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.855738 2545 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-online-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:12:28.855771 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.855768 2545 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-offline-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:12:28.855771 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.855776 2545 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-ui-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:12:28.856098 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.855784 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbchg\" (UniqueName: \"kubernetes.io/projected/17dd1591-53a4-4245-a683-76cfaf4e3f53-kube-api-access-dbchg\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:12:28.856098 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.855793 2545 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17dd1591-53a4-4245-a683-76cfaf4e3f53-feast-data\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:12:28.856098 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.855801 2545 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17dd1591-53a4-4245-a683-76cfaf4e3f53-registry-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:12:28.906900 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.906866 2545 generic.go:356] "Generic (PLEG): container finished" podID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerID="bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865" exitCode=137 Jun 10 11:12:28.906900 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.906894 2545 generic.go:356] "Generic (PLEG): container finished" podID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerID="b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8" exitCode=137 Jun 10 11:12:28.907165 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.906935 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" Jun 10 11:12:28.907165 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.906955 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerDied","Data":"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865"} Jun 10 11:12:28.907165 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.907016 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerDied","Data":"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8"} Jun 10 11:12:28.907165 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.907032 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz" event={"ID":"17dd1591-53a4-4245-a683-76cfaf4e3f53","Type":"ContainerDied","Data":"9868f2196d473506ca9a646d3d5407e42ef246bc11004be35e92c57004862116"} Jun 10 11:12:28.907165 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.907051 2545 scope.go:117] "RemoveContainer" containerID="2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521" Jun 10 11:12:28.915452 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.915419 2545 scope.go:117] "RemoveContainer" containerID="bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865" Jun 10 11:12:28.924332 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.924232 2545 scope.go:117] "RemoveContainer" containerID="bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f" Jun 10 11:12:28.930440 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.930414 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz"] Jun 10 11:12:28.932701 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.932672 2545 scope.go:117] "RemoveContainer" containerID="b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8" Jun 10 11:12:28.934393 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.934375 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-6k5mz"] Jun 10 11:12:28.939795 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.939778 2545 scope.go:117] "RemoveContainer" containerID="a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3" Jun 10 11:12:28.946558 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.946530 2545 scope.go:117] "RemoveContainer" containerID="d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e" Jun 10 11:12:28.953043 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.953026 2545 scope.go:117] "RemoveContainer" containerID="2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521" Jun 10 11:12:28.953303 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:12:28.953281 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521\": container with ID starting with 2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521 not found: ID does not exist" containerID="2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521" Jun 10 11:12:28.953350 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.953312 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521"} err="failed to get container status \"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521\": rpc error: code = NotFound desc = could not find container \"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521\": container with ID starting with 2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521 not found: ID does not exist" Jun 10 11:12:28.953350 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.953331 2545 scope.go:117] "RemoveContainer" containerID="bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865" Jun 10 11:12:28.953580 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:12:28.953563 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865\": container with ID starting with bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865 not found: ID does not exist" containerID="bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865" Jun 10 11:12:28.953626 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.953585 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865"} err="failed to get container status \"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865\": rpc error: code = NotFound desc = could not find container \"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865\": container with ID starting with bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865 not found: ID does not exist" Jun 10 11:12:28.953626 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.953601 2545 scope.go:117] "RemoveContainer" containerID="bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f" Jun 10 11:12:28.953811 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:12:28.953796 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f\": container with ID starting with bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f not found: ID does not exist" containerID="bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f" Jun 10 11:12:28.953854 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.953816 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f"} err="failed to get container status \"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f\": rpc error: code = NotFound desc = could not find container \"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f\": container with ID starting with bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f not found: ID does not exist" Jun 10 11:12:28.953854 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.953829 2545 scope.go:117] "RemoveContainer" containerID="b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8" Jun 10 11:12:28.954094 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:12:28.954075 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8\": container with ID starting with b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8 not found: ID does not exist" containerID="b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8" Jun 10 11:12:28.954142 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954101 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8"} err="failed to get container status \"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8\": rpc error: code = NotFound desc = could not find container \"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8\": container with ID starting with b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8 not found: ID does not exist" Jun 10 11:12:28.954142 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954119 2545 scope.go:117] "RemoveContainer" containerID="a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3" Jun 10 11:12:28.954387 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:12:28.954367 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3\": container with ID starting with a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3 not found: ID does not exist" containerID="a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3" Jun 10 11:12:28.954462 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954389 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3"} err="failed to get container status \"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3\": rpc error: code = NotFound desc = could not find container \"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3\": container with ID starting with a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3 not found: ID does not exist" Jun 10 11:12:28.954462 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954403 2545 scope.go:117] "RemoveContainer" containerID="d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e" Jun 10 11:12:28.954619 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:12:28.954601 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e\": container with ID starting with d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e not found: ID does not exist" containerID="d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e" Jun 10 11:12:28.954664 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954626 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e"} err="failed to get container status \"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e\": rpc error: code = NotFound desc = could not find container \"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e\": container with ID starting with d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e not found: ID does not exist" Jun 10 11:12:28.954664 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954640 2545 scope.go:117] "RemoveContainer" containerID="2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521" Jun 10 11:12:28.954872 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954852 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521"} err="failed to get container status \"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521\": rpc error: code = NotFound desc = could not find container \"2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521\": container with ID starting with 2962d5773383701f0fb51ea26c35e47babd6a009d3e7b9e12e85eae32e9df521 not found: ID does not exist" Jun 10 11:12:28.954872 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.954870 2545 scope.go:117] "RemoveContainer" containerID="bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865" Jun 10 11:12:28.955109 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955092 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865"} err="failed to get container status \"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865\": rpc error: code = NotFound desc = could not find container \"bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865\": container with ID starting with bb320b3f74d5716b671e9f633f78f4faaaa7e82635431b1936826d6182c5d865 not found: ID does not exist" Jun 10 11:12:28.955167 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955111 2545 scope.go:117] "RemoveContainer" containerID="bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f" Jun 10 11:12:28.955329 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955312 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f"} err="failed to get container status \"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f\": rpc error: code = NotFound desc = could not find container \"bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f\": container with ID starting with bbec3242f488080e121d3826fb67d1bad9057f43d0e07478ce6387fe9d72452f not found: ID does not exist" Jun 10 11:12:28.955369 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955329 2545 scope.go:117] "RemoveContainer" containerID="b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8" Jun 10 11:12:28.955540 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955521 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8"} err="failed to get container status \"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8\": rpc error: code = NotFound desc = could not find container \"b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8\": container with ID starting with b22494903fbc99a6f0d096f6050be0db695410b9236e17cc06884eda7bd8f8e8 not found: ID does not exist" Jun 10 11:12:28.955578 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955542 2545 scope.go:117] "RemoveContainer" containerID="a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3" Jun 10 11:12:28.955765 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955748 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3"} err="failed to get container status \"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3\": rpc error: code = NotFound desc = could not find container \"a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3\": container with ID starting with a1d1604f0e90d72d3a5cf991263dc9748782c08e2bfa08b67a042f180b2fd6e3 not found: ID does not exist" Jun 10 11:12:28.955810 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955767 2545 scope.go:117] "RemoveContainer" containerID="d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e" Jun 10 11:12:28.955999 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:28.955977 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e"} err="failed to get container status \"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e\": rpc error: code = NotFound desc = could not find container \"d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e\": container with ID starting with d50ec9019c017242400563f3352fb100f53bdc88a384077f9dabeac30729069e not found: ID does not exist" Jun 10 11:12:30.781651 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:12:30.781619 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" path="/var/lib/kubelet/pods/17dd1591-53a4-4245-a683-76cfaf4e3f53/volumes" Jun 10 11:13:19.802637 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802556 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k"] Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802835 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="feast-apply" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802846 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="feast-apply" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802858 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802863 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802872 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="offline" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802877 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="offline" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802884 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="registry" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802889 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="registry" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802896 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802901 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802907 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="feast-init" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802912 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="feast-init" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802958 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="online" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802981 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="ui" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802989 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="registry" Jun 10 11:13:19.803102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.802995 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="17dd1591-53a4-4245-a683-76cfaf4e3f53" containerName="offline" Jun 10 11:13:19.806076 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.806061 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.809449 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.809429 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jun 10 11:13:19.809550 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.809446 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jun 10 11:13:19.809550 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.809521 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-szsdn\"" Jun 10 11:13:19.809550 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.809525 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jun 10 11:13:19.809763 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.809746 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jun 10 11:13:19.809826 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.809788 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jun 10 11:13:19.810552 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.810537 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jun 10 11:13:19.821793 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.821774 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k"] Jun 10 11:13:19.864564 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.864543 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd182da-3cd4-43f2-a90d-29e44ca14933-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.864679 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.864570 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pff2f\" (UniqueName: \"kubernetes.io/projected/2cd182da-3cd4-43f2-a90d-29e44ca14933-kube-api-access-pff2f\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.864679 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.864617 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2cd182da-3cd4-43f2-a90d-29e44ca14933-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.864790 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.864728 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.864847 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.864802 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.864847 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.864832 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965168 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965141 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965266 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965170 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965266 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965201 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd182da-3cd4-43f2-a90d-29e44ca14933-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965266 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965216 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pff2f\" (UniqueName: \"kubernetes.io/projected/2cd182da-3cd4-43f2-a90d-29e44ca14933-kube-api-access-pff2f\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965266 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965244 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2cd182da-3cd4-43f2-a90d-29e44ca14933-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965424 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:13:19.965269 2545 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 10 11:13:19.965424 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965289 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965424 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:13:19.965326 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls podName:2cd182da-3cd4-43f2-a90d-29e44ca14933 nodeName:}" failed. No retries permitted until 2026-06-10 11:13:20.465306636 +0000 UTC m=+1702.114239170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls") pod "feast-simple-feast-remote-setup-669bcf45fc-htx5k" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933") : secret "feast-simple-feast-remote-setup-ui-tls" not found Jun 10 11:13:19.965424 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:13:19.965268 2545 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jun 10 11:13:19.965424 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:13:19.965354 2545 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 10 11:13:19.965424 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:13:19.965420 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls podName:2cd182da-3cd4-43f2-a90d-29e44ca14933 nodeName:}" failed. No retries permitted until 2026-06-10 11:13:20.465399977 +0000 UTC m=+1702.114332512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls") pod "feast-simple-feast-remote-setup-669bcf45fc-htx5k" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933") : secret "feast-simple-feast-remote-setup-online-tls" not found Jun 10 11:13:19.965645 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:13:19.965440 2545 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls podName:2cd182da-3cd4-43f2-a90d-29e44ca14933 nodeName:}" failed. No retries permitted until 2026-06-10 11:13:20.46543116 +0000 UTC m=+1702.114363685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls") pod "feast-simple-feast-remote-setup-669bcf45fc-htx5k" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933") : secret "feast-simple-feast-remote-setup-offline-tls" not found Jun 10 11:13:19.965645 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965531 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd182da-3cd4-43f2-a90d-29e44ca14933-feast-data\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.965785 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.965769 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2cd182da-3cd4-43f2-a90d-29e44ca14933-registry-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:19.978645 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:19.978615 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pff2f\" (UniqueName: \"kubernetes.io/projected/2cd182da-3cd4-43f2-a90d-29e44ca14933-kube-api-access-pff2f\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.469703 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.469670 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.469907 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.469721 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.469907 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.469740 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.472227 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.472181 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.472227 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.472204 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.472405 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.472274 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls\") pod \"feast-simple-feast-remote-setup-669bcf45fc-htx5k\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.715780 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.715735 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:20.857622 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:20.857596 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k"] Jun 10 11:13:20.858819 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:13:20.858792 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cd182da_3cd4_43f2_a90d_29e44ca14933.slice/crio-bda94c8822e75b713aef82cf098fd71fe5407d3c70a5625a2741ed1d264bf5bc WatchSource:0}: Error finding container bda94c8822e75b713aef82cf098fd71fe5407d3c70a5625a2741ed1d264bf5bc: Status 404 returned error can't find the container with id bda94c8822e75b713aef82cf098fd71fe5407d3c70a5625a2741ed1d264bf5bc Jun 10 11:13:21.103012 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:21.102168 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerStarted","Data":"08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641"} Jun 10 11:13:21.103012 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:21.102221 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerStarted","Data":"bda94c8822e75b713aef82cf098fd71fe5407d3c70a5625a2741ed1d264bf5bc"} Jun 10 11:13:26.120764 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:26.120732 2545 generic.go:356] "Generic (PLEG): container finished" podID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerID="08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641" exitCode=0 Jun 10 11:13:26.121137 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:26.120813 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerDied","Data":"08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641"} Jun 10 11:13:27.126596 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:27.126561 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerStarted","Data":"986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049"} Jun 10 11:13:32.148392 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:32.148356 2545 generic.go:356] "Generic (PLEG): container finished" podID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerID="986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049" exitCode=0 Jun 10 11:13:32.148756 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:32.148427 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerDied","Data":"986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049"} Jun 10 11:13:33.155932 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:33.155894 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerStarted","Data":"a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e"} Jun 10 11:13:33.156320 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:33.155950 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerStarted","Data":"2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936"} Jun 10 11:13:33.156320 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:33.155985 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerStarted","Data":"c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6"} Jun 10 11:13:33.190482 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:33.190425 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podStartSLOduration=14.190408823 podStartE2EDuration="14.190408823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-10 11:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:13:33.188294956 +0000 UTC m=+1714.837227498" watchObservedRunningTime="2026-06-10 11:13:33.190408823 +0000 UTC m=+1714.839341363" Jun 10 11:13:35.716246 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:35.716209 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:35.716741 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:35.716290 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:35.716741 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:35.716309 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:35.717814 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:35.717771 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jun 10 11:13:35.717947 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:35.717806 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Jun 10 11:13:35.717947 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:35.717828 2545 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jun 10 11:13:38.716708 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:38.716677 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:38.717166 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:38.717148 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:38.717221 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:38.717187 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:38.717495 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:38.717479 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:38.720907 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:38.720892 2545 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:39.182615 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:39.182585 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:39.182768 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:39.182631 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:39.183127 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:39.183101 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:13:39.185633 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:13:39.185615 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:14:40.842636 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.842560 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k"] Jun 10 11:14:40.843173 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.842802 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" containerID="cri-o://c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6" gracePeriod=30 Jun 10 11:14:40.843173 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.842850 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="offline" containerID="cri-o://2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936" gracePeriod=30 Jun 10 11:14:40.843173 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.842850 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" containerID="cri-o://a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e" gracePeriod=30 Jun 10 11:14:40.973380 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.973348 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m"] Jun 10 11:14:40.973812 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.973594 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="registry" containerID="cri-o://12b22499161d63bae1c9c36146cc85ddf40be81372a28c8915ad38f959cc6147" gracePeriod=30 Jun 10 11:14:40.973812 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.973632 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="offline" containerID="cri-o://92bb69e1db0515e5a7e84c02132ab56ce7f251d8ca156df62ea5e6e377030cfe" gracePeriod=30 Jun 10 11:14:40.973812 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.973639 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" containerID="cri-o://7ef6865ffc49d947ae1b150687f9e1196ff9c73d4c92cf7350e176678809e4a7" gracePeriod=30 Jun 10 11:14:40.973812 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:40.973693 2545 kuberuntime_container.go:872] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" containerID="cri-o://a94b7055061f8c1d7af3da4bad56fa587fac989c11e0e891fc60f85de218ade8" gracePeriod=30 Jun 10 11:14:41.415556 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:41.415513 2545 generic.go:356] "Generic (PLEG): container finished" podID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerID="a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e" exitCode=0 Jun 10 11:14:41.415556 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:41.415544 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerDied","Data":"a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e"} Jun 10 11:14:42.422752 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:42.422722 2545 generic.go:356] "Generic (PLEG): container finished" podID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerID="7ef6865ffc49d947ae1b150687f9e1196ff9c73d4c92cf7350e176678809e4a7" exitCode=0 Jun 10 11:14:42.422752 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:42.422745 2545 generic.go:356] "Generic (PLEG): container finished" podID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerID="a94b7055061f8c1d7af3da4bad56fa587fac989c11e0e891fc60f85de218ade8" exitCode=0 Jun 10 11:14:42.423239 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:42.422793 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerDied","Data":"7ef6865ffc49d947ae1b150687f9e1196ff9c73d4c92cf7350e176678809e4a7"} Jun 10 11:14:42.423239 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:42.422844 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerDied","Data":"a94b7055061f8c1d7af3da4bad56fa587fac989c11e0e891fc60f85de218ade8"} Jun 10 11:14:42.425383 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:42.425356 2545 generic.go:356] "Generic (PLEG): container finished" podID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerID="c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6" exitCode=0 Jun 10 11:14:42.425509 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:42.425405 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerDied","Data":"c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6"} Jun 10 11:14:46.771407 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:46.771358 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Jun 10 11:14:46.858154 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:46.858107 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Jun 10 11:14:48.718293 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:48.718251 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jun 10 11:14:49.183500 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:49.183459 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jun 10 11:14:56.771332 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:56.771285 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Jun 10 11:14:56.858035 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:56.857995 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Jun 10 11:14:58.717186 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:58.717146 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jun 10 11:14:59.183437 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:14:59.183389 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jun 10 11:15:06.770554 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:06.770505 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Jun 10 11:15:06.770944 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:06.770664 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:15:06.857426 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:06.857389 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Jun 10 11:15:06.857603 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:06.857503 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:15:08.717256 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:08.717205 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jun 10 11:15:08.717677 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:08.717371 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:15:09.182948 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:09.182909 2545 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jun 10 11:15:09.183165 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:09.183084 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:15:11.060525 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.060504 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:15:11.078579 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.078557 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls\") pod \"2cd182da-3cd4-43f2-a90d-29e44ca14933\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " Jun 10 11:15:11.078670 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.078596 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2cd182da-3cd4-43f2-a90d-29e44ca14933-registry-tls\") pod \"2cd182da-3cd4-43f2-a90d-29e44ca14933\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " Jun 10 11:15:11.078670 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.078618 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls\") pod \"2cd182da-3cd4-43f2-a90d-29e44ca14933\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " Jun 10 11:15:11.078670 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.078651 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls\") pod \"2cd182da-3cd4-43f2-a90d-29e44ca14933\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " Jun 10 11:15:11.078821 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.078688 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pff2f\" (UniqueName: \"kubernetes.io/projected/2cd182da-3cd4-43f2-a90d-29e44ca14933-kube-api-access-pff2f\") pod \"2cd182da-3cd4-43f2-a90d-29e44ca14933\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " Jun 10 11:15:11.078821 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.078739 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd182da-3cd4-43f2-a90d-29e44ca14933-feast-data\") pod \"2cd182da-3cd4-43f2-a90d-29e44ca14933\" (UID: \"2cd182da-3cd4-43f2-a90d-29e44ca14933\") " Jun 10 11:15:11.079073 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.079044 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd182da-3cd4-43f2-a90d-29e44ca14933-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2cd182da-3cd4-43f2-a90d-29e44ca14933" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 10 11:15:11.079270 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.079245 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd182da-3cd4-43f2-a90d-29e44ca14933-feast-data" (OuterVolumeSpecName: "feast-data") pod "2cd182da-3cd4-43f2-a90d-29e44ca14933" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:15:11.080723 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.080702 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2cd182da-3cd4-43f2-a90d-29e44ca14933" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:15:11.080812 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.080774 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls" (OuterVolumeSpecName: "online-tls") pod "2cd182da-3cd4-43f2-a90d-29e44ca14933" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:15:11.081105 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.081076 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2cd182da-3cd4-43f2-a90d-29e44ca14933" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:15:11.081105 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.081089 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd182da-3cd4-43f2-a90d-29e44ca14933-kube-api-access-pff2f" (OuterVolumeSpecName: "kube-api-access-pff2f") pod "2cd182da-3cd4-43f2-a90d-29e44ca14933" (UID: "2cd182da-3cd4-43f2-a90d-29e44ca14933"). InnerVolumeSpecName "kube-api-access-pff2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:15:11.179306 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.179276 2545 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-ui-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.179306 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.179302 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pff2f\" (UniqueName: \"kubernetes.io/projected/2cd182da-3cd4-43f2-a90d-29e44ca14933-kube-api-access-pff2f\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.179306 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.179311 2545 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd182da-3cd4-43f2-a90d-29e44ca14933-feast-data\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.179534 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.179322 2545 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-offline-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.179534 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.179330 2545 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2cd182da-3cd4-43f2-a90d-29e44ca14933-registry-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.179534 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.179357 2545 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2cd182da-3cd4-43f2-a90d-29e44ca14933-online-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.542159 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.542130 2545 generic.go:356] "Generic (PLEG): container finished" podID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerID="92bb69e1db0515e5a7e84c02132ab56ce7f251d8ca156df62ea5e6e377030cfe" exitCode=137 Jun 10 11:15:11.542159 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.542157 2545 generic.go:356] "Generic (PLEG): container finished" podID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerID="12b22499161d63bae1c9c36146cc85ddf40be81372a28c8915ad38f959cc6147" exitCode=137 Jun 10 11:15:11.542344 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.542202 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerDied","Data":"92bb69e1db0515e5a7e84c02132ab56ce7f251d8ca156df62ea5e6e377030cfe"} Jun 10 11:15:11.542344 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.542248 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerDied","Data":"12b22499161d63bae1c9c36146cc85ddf40be81372a28c8915ad38f959cc6147"} Jun 10 11:15:11.544757 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.544738 2545 generic.go:356] "Generic (PLEG): container finished" podID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerID="2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936" exitCode=137 Jun 10 11:15:11.544871 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.544800 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" Jun 10 11:15:11.544871 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.544822 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerDied","Data":"2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936"} Jun 10 11:15:11.545000 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.544881 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k" event={"ID":"2cd182da-3cd4-43f2-a90d-29e44ca14933","Type":"ContainerDied","Data":"bda94c8822e75b713aef82cf098fd71fe5407d3c70a5625a2741ed1d264bf5bc"} Jun 10 11:15:11.545000 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.544909 2545 scope.go:117] "RemoveContainer" containerID="a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e" Jun 10 11:15:11.554280 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.554258 2545 scope.go:117] "RemoveContainer" containerID="2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936" Jun 10 11:15:11.563818 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.563799 2545 scope.go:117] "RemoveContainer" containerID="c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6" Jun 10 11:15:11.570739 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.570704 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k"] Jun 10 11:15:11.572643 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.572620 2545 scope.go:117] "RemoveContainer" containerID="986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049" Jun 10 11:15:11.577083 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.577055 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-669bcf45fc-htx5k"] Jun 10 11:15:11.580274 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.580248 2545 scope.go:117] "RemoveContainer" containerID="08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641" Jun 10 11:15:11.587282 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.587257 2545 scope.go:117] "RemoveContainer" containerID="a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e" Jun 10 11:15:11.587538 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:15:11.587521 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e\": container with ID starting with a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e not found: ID does not exist" containerID="a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e" Jun 10 11:15:11.587596 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.587546 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e"} err="failed to get container status \"a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e\": rpc error: code = NotFound desc = could not find container \"a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e\": container with ID starting with a76fe92a92eba7d69bc706499a790d189ed37b6163102ffc969c18d2cce8e94e not found: ID does not exist" Jun 10 11:15:11.587596 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.587563 2545 scope.go:117] "RemoveContainer" containerID="2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936" Jun 10 11:15:11.587842 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:15:11.587813 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936\": container with ID starting with 2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936 not found: ID does not exist" containerID="2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936" Jun 10 11:15:11.587925 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.587852 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936"} err="failed to get container status \"2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936\": rpc error: code = NotFound desc = could not find container \"2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936\": container with ID starting with 2fc785f7ec27f9464abcb519132a80d58465ec854ec16ee2314c046f27f87936 not found: ID does not exist" Jun 10 11:15:11.587925 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.587875 2545 scope.go:117] "RemoveContainer" containerID="c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6" Jun 10 11:15:11.588141 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:15:11.588124 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6\": container with ID starting with c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6 not found: ID does not exist" containerID="c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6" Jun 10 11:15:11.588187 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.588146 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6"} err="failed to get container status \"c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6\": rpc error: code = NotFound desc = could not find container \"c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6\": container with ID starting with c90c4d2b849160594560f6c0d2c8c875036420e1c1587500f7feeb70cce37bb6 not found: ID does not exist" Jun 10 11:15:11.588187 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.588165 2545 scope.go:117] "RemoveContainer" containerID="986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049" Jun 10 11:15:11.588390 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:15:11.588373 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049\": container with ID starting with 986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049 not found: ID does not exist" containerID="986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049" Jun 10 11:15:11.588437 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.588394 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049"} err="failed to get container status \"986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049\": rpc error: code = NotFound desc = could not find container \"986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049\": container with ID starting with 986f6fdfc6ef1fa84a1cced25031fa5087a9646b7cc66e6fc14de8943f506049 not found: ID does not exist" Jun 10 11:15:11.588437 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.588410 2545 scope.go:117] "RemoveContainer" containerID="08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641" Jun 10 11:15:11.588614 ip-10-0-128-191 kubenswrapper[2545]: E0610 11:15:11.588596 2545 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641\": container with ID starting with 08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641 not found: ID does not exist" containerID="08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641" Jun 10 11:15:11.588655 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.588619 2545 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641"} err="failed to get container status \"08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641\": rpc error: code = NotFound desc = could not find container \"08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641\": container with ID starting with 08d422a0f99d6040287c1d6c1068dcdc4547687189899cc95f6a0028156da641 not found: ID does not exist" Jun 10 11:15:11.618032 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.618009 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:15:11.681068 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.681035 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-offline-tls\") pod \"fb0ab609-8eb4-405c-8168-1be8cce83065\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " Jun 10 11:15:11.681301 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.681080 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-registry-tls\") pod \"fb0ab609-8eb4-405c-8168-1be8cce83065\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " Jun 10 11:15:11.681301 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.681104 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-ui-tls\") pod \"fb0ab609-8eb4-405c-8168-1be8cce83065\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " Jun 10 11:15:11.681301 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.681130 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ab609-8eb4-405c-8168-1be8cce83065-feast-data\") pod \"fb0ab609-8eb4-405c-8168-1be8cce83065\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " Jun 10 11:15:11.681301 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.681284 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-online-tls\") pod \"fb0ab609-8eb4-405c-8168-1be8cce83065\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " Jun 10 11:15:11.681513 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.681393 2545 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz8gp\" (UniqueName: \"kubernetes.io/projected/fb0ab609-8eb4-405c-8168-1be8cce83065-kube-api-access-lz8gp\") pod \"fb0ab609-8eb4-405c-8168-1be8cce83065\" (UID: \"fb0ab609-8eb4-405c-8168-1be8cce83065\") " Jun 10 11:15:11.681653 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.681628 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0ab609-8eb4-405c-8168-1be8cce83065-feast-data" (OuterVolumeSpecName: "feast-data") pod "fb0ab609-8eb4-405c-8168-1be8cce83065" (UID: "fb0ab609-8eb4-405c-8168-1be8cce83065"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 10 11:15:11.683179 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.683149 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "fb0ab609-8eb4-405c-8168-1be8cce83065" (UID: "fb0ab609-8eb4-405c-8168-1be8cce83065"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:15:11.683310 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.683272 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-online-tls" (OuterVolumeSpecName: "online-tls") pod "fb0ab609-8eb4-405c-8168-1be8cce83065" (UID: "fb0ab609-8eb4-405c-8168-1be8cce83065"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:15:11.683393 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.683351 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0ab609-8eb4-405c-8168-1be8cce83065-kube-api-access-lz8gp" (OuterVolumeSpecName: "kube-api-access-lz8gp") pod "fb0ab609-8eb4-405c-8168-1be8cce83065" (UID: "fb0ab609-8eb4-405c-8168-1be8cce83065"). InnerVolumeSpecName "kube-api-access-lz8gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 10 11:15:11.683595 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.683575 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fb0ab609-8eb4-405c-8168-1be8cce83065" (UID: "fb0ab609-8eb4-405c-8168-1be8cce83065"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:15:11.683690 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.683673 2545 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fb0ab609-8eb4-405c-8168-1be8cce83065" (UID: "fb0ab609-8eb4-405c-8168-1be8cce83065"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 10 11:15:11.782509 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.782398 2545 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-online-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.782509 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.782440 2545 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8gp\" (UniqueName: \"kubernetes.io/projected/fb0ab609-8eb4-405c-8168-1be8cce83065-kube-api-access-lz8gp\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.782509 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.782455 2545 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-offline-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.782509 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.782466 2545 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-registry-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.782509 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.782476 2545 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fb0ab609-8eb4-405c-8168-1be8cce83065-ui-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:11.782509 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:11.782485 2545 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0ab609-8eb4-405c-8168-1be8cce83065-feast-data\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\"" Jun 10 11:15:12.551109 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.551075 2545 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" Jun 10 11:15:12.551551 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.551073 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m" event={"ID":"fb0ab609-8eb4-405c-8168-1be8cce83065","Type":"ContainerDied","Data":"8fdf1446f2ccb4d9edf9206dd50c9a101e3333003b4f1632ed27ccd8cc634cc7"} Jun 10 11:15:12.551551 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.551203 2545 scope.go:117] "RemoveContainer" containerID="7ef6865ffc49d947ae1b150687f9e1196ff9c73d4c92cf7350e176678809e4a7" Jun 10 11:15:12.559725 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.559707 2545 scope.go:117] "RemoveContainer" containerID="92bb69e1db0515e5a7e84c02132ab56ce7f251d8ca156df62ea5e6e377030cfe" Jun 10 11:15:12.566827 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.566809 2545 scope.go:117] "RemoveContainer" containerID="a94b7055061f8c1d7af3da4bad56fa587fac989c11e0e891fc60f85de218ade8" Jun 10 11:15:12.573872 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.573830 2545 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m"] Jun 10 11:15:12.574188 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.574167 2545 scope.go:117] "RemoveContainer" containerID="12b22499161d63bae1c9c36146cc85ddf40be81372a28c8915ad38f959cc6147" Jun 10 11:15:12.584131 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.584104 2545 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-75c64c888b-q7g8m"] Jun 10 11:15:12.585147 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.585129 2545 scope.go:117] "RemoveContainer" containerID="841c97f1337cfb9e402a3228c71746f6f0f553e72d024b5ad2931a1b65f69e80" Jun 10 11:15:12.592026 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.592003 2545 scope.go:117] "RemoveContainer" containerID="5e2b6b4e55c41bd6aaf5b17ce5b86b3eb9099c06a14b88842e49f8e5e0d5dc13" Jun 10 11:15:12.781099 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.781065 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" path="/var/lib/kubelet/pods/2cd182da-3cd4-43f2-a90d-29e44ca14933/volumes" Jun 10 11:15:12.781609 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:12.781596 2545 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" path="/var/lib/kubelet/pods/fb0ab609-8eb4-405c-8168-1be8cce83065/volumes" Jun 10 11:15:50.251738 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:50.251708 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-47m9g_ff371f6a-32fb-42d5-96f8-d1f6558bf6fa/global-pull-secret-syncer/0.log" Jun 10 11:15:50.454959 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:50.454929 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ql89z_637ed0ce-8721-46e2-8c84-bd306db605c0/konnectivity-agent/0.log" Jun 10 11:15:50.478019 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:50.477983 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-191.ec2.internal_75bf6497d56e225e5bd01fc8d3fdd351/haproxy/0.log" Jun 10 11:15:54.341189 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.341149 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36462199-1c62-4df5-bb91-c7b0b58cdce4/alertmanager/0.log" Jun 10 11:15:54.369497 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.369464 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36462199-1c62-4df5-bb91-c7b0b58cdce4/config-reloader/0.log" Jun 10 11:15:54.394649 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.394624 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36462199-1c62-4df5-bb91-c7b0b58cdce4/kube-rbac-proxy-web/0.log" Jun 10 11:15:54.425354 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.425323 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36462199-1c62-4df5-bb91-c7b0b58cdce4/kube-rbac-proxy/0.log" Jun 10 11:15:54.453543 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.453506 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36462199-1c62-4df5-bb91-c7b0b58cdce4/kube-rbac-proxy-metric/0.log" Jun 10 11:15:54.482817 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.482784 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36462199-1c62-4df5-bb91-c7b0b58cdce4/prom-label-proxy/0.log" Jun 10 11:15:54.514995 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.514949 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_36462199-1c62-4df5-bb91-c7b0b58cdce4/init-config-reloader/0.log" Jun 10 11:15:54.717362 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.717333 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-76b68cc874-m9c65_fabe843c-fbeb-4b61-ba3c-424fafe0a1b6/monitoring-plugin/0.log" Jun 10 11:15:54.754270 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.754235 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnwsx_c80e2331-c8d3-4bf5-8c4d-6047c3652f93/node-exporter/0.log" Jun 10 11:15:54.780159 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.780121 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnwsx_c80e2331-c8d3-4bf5-8c4d-6047c3652f93/kube-rbac-proxy/0.log" Jun 10 11:15:54.808912 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:54.808886 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mnwsx_c80e2331-c8d3-4bf5-8c4d-6047c3652f93/init-textfile/0.log" Jun 10 11:15:55.432565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:55.432535 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-584f4f48dc-gbxgf_904a7605-4da6-4447-882b-a20cea57ab98/telemeter-client/0.log" Jun 10 11:15:55.470556 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:55.470530 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-584f4f48dc-gbxgf_904a7605-4da6-4447-882b-a20cea57ab98/reload/0.log" Jun 10 11:15:55.511129 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:55.511099 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-584f4f48dc-gbxgf_904a7605-4da6-4447-882b-a20cea57ab98/kube-rbac-proxy/0.log" Jun 10 11:15:57.659210 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659180 2545 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2"] Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659463 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="feast-init" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659474 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="feast-init" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659484 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="feast-init" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659490 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="feast-init" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659498 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="registry" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659503 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="registry" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659509 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659513 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659520 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659526 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659532 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659538 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659544 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="feast-apply" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659549 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="feast-apply" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659557 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="offline" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659561 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="offline" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659567 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="feast-apply" Jun 10 11:15:57.659565 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659573 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="feast-apply" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659581 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="offline" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659586 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="offline" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659593 2545 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659597 2545 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659636 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="registry" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659644 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="ui" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659651 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="online" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659657 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="offline" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659662 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd182da-3cd4-43f2-a90d-29e44ca14933" containerName="online" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659666 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="offline" Jun 10 11:15:57.660254 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.659670 2545 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0ab609-8eb4-405c-8168-1be8cce83065" containerName="ui" Jun 10 11:15:57.662568 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.662548 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.666083 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.666066 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g7cc7\"/\"kube-root-ca.crt\"" Jun 10 11:15:57.666198 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.666147 2545 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g7cc7\"/\"default-dockercfg-8fnkx\"" Jun 10 11:15:57.667454 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.667438 2545 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g7cc7\"/\"openshift-service-ca.crt\"" Jun 10 11:15:57.678102 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.678077 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2"] Jun 10 11:15:57.718424 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.718396 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bd9k\" (UniqueName: \"kubernetes.io/projected/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-kube-api-access-6bd9k\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.718424 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.718425 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-lib-modules\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.718615 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.718443 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-proc\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.718615 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.718464 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-podres\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.718615 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.718555 2545 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-sys\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.724179 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.724152 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-596f78d89b-89nfr_54075fc2-edfe-4638-bb25-4988461b43b1/console/0.log" Jun 10 11:15:57.765016 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.764989 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6657875f97-g4jrs_fdf31123-573d-4704-b464-1c1cabd7b5e9/download-server/0.log" Jun 10 11:15:57.819096 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819069 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-proc\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819213 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819103 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-podres\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819213 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819178 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-sys\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819213 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819192 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-proc\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819334 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819234 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bd9k\" (UniqueName: \"kubernetes.io/projected/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-kube-api-access-6bd9k\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819334 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819267 2545 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-lib-modules\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819334 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819241 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-sys\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819334 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819297 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-podres\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.819519 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.819370 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-lib-modules\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.828596 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.828573 2545 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bd9k\" (UniqueName: \"kubernetes.io/projected/22ff3143-b0aa-43e4-9d40-b9aa649ce1ba-kube-api-access-6bd9k\") pod \"perf-node-gather-daemonset-7znw2\" (UID: \"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba\") " pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:57.971625 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:57.971530 2545 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:58.094931 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:58.094903 2545 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2"] Jun 10 11:15:58.097078 ip-10-0-128-191 kubenswrapper[2545]: W0610 11:15:58.097046 2545 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod22ff3143_b0aa_43e4_9d40_b9aa649ce1ba.slice/crio-3d046b8a8f4550867e1f7789fe051a0410e13b72729983ccfff50e79ce7af393 WatchSource:0}: Error finding container 3d046b8a8f4550867e1f7789fe051a0410e13b72729983ccfff50e79ce7af393: Status 404 returned error can't find the container with id 3d046b8a8f4550867e1f7789fe051a0410e13b72729983ccfff50e79ce7af393 Jun 10 11:15:58.718479 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:58.718446 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" event={"ID":"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba","Type":"ContainerStarted","Data":"e437cf222a494c183e859622233e3b858e67cfb5be61b8a59f7061074d9b4933"} Jun 10 11:15:58.718927 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:58.718520 2545 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" event={"ID":"22ff3143-b0aa-43e4-9d40-b9aa649ce1ba","Type":"ContainerStarted","Data":"3d046b8a8f4550867e1f7789fe051a0410e13b72729983ccfff50e79ce7af393"} Jun 10 11:15:58.718927 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:58.718590 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:15:58.749556 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:58.749511 2545 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" podStartSLOduration=1.749497358 podStartE2EDuration="1.749497358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-10 11:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-10 11:15:58.744450614 +0000 UTC m=+1860.393383165" watchObservedRunningTime="2026-06-10 11:15:58.749497358 +0000 UTC m=+1860.398429898" Jun 10 11:15:59.262855 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:59.262827 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tc8c9_252cfbef-2c8f-4d25-971d-cf008884454d/dns/0.log" Jun 10 11:15:59.290717 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:59.290692 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tc8c9_252cfbef-2c8f-4d25-971d-cf008884454d/kube-rbac-proxy/0.log" Jun 10 11:15:59.427792 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:59.427759 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p4l5s_c989fdb9-e253-413c-b23f-8d6c1389d16c/dns-node-resolver/0.log" Jun 10 11:15:59.900946 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:15:59.900915 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jgp8h_64be67c0-7c44-4334-b8d7-29aa305f5fd5/node-ca/0.log" Jun 10 11:16:01.071413 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:01.071387 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dqngk_8e608297-9570-4900-bba2-7f7a280d9779/serve-healthcheck-canary/0.log" Jun 10 11:16:01.685523 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:01.685495 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sx4p2_ba682d21-f1a8-4129-b14a-dd72ceb88452/kube-rbac-proxy/0.log" Jun 10 11:16:01.714100 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:01.714069 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sx4p2_ba682d21-f1a8-4129-b14a-dd72ceb88452/exporter/0.log" Jun 10 11:16:01.744479 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:01.744438 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sx4p2_ba682d21-f1a8-4129-b14a-dd72ceb88452/extractor/0.log" Jun 10 11:16:04.730981 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:04.730932 2545 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g7cc7/perf-node-gather-daemonset-7znw2" Jun 10 11:16:07.611590 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:07.611564 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxkhd_db8e86ee-d321-4fdc-be9a-495560a2f399/kube-multus-additional-cni-plugins/0.log" Jun 10 11:16:07.635794 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:07.635768 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxkhd_db8e86ee-d321-4fdc-be9a-495560a2f399/egress-router-binary-copy/0.log" Jun 10 11:16:07.661424 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:07.661400 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxkhd_db8e86ee-d321-4fdc-be9a-495560a2f399/cni-plugins/0.log" Jun 10 11:16:07.686887 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:07.686855 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxkhd_db8e86ee-d321-4fdc-be9a-495560a2f399/bond-cni-plugin/0.log" Jun 10 11:16:07.712400 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:07.712377 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxkhd_db8e86ee-d321-4fdc-be9a-495560a2f399/routeoverride-cni/0.log" Jun 10 11:16:07.737746 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:07.737725 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxkhd_db8e86ee-d321-4fdc-be9a-495560a2f399/whereabouts-cni-bincopy/0.log" Jun 10 11:16:07.777216 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:07.777192 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cxkhd_db8e86ee-d321-4fdc-be9a-495560a2f399/whereabouts-cni/0.log" Jun 10 11:16:08.340589 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:08.340557 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h4kch_b01aab23-af40-42ef-914f-2ca0ea129769/kube-multus/0.log" Jun 10 11:16:08.367992 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:08.367942 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6xj9p_452be7ad-2d9a-4448-899a-d5e68cc10f2e/network-metrics-daemon/0.log" Jun 10 11:16:08.398153 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:08.398080 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6xj9p_452be7ad-2d9a-4448-899a-d5e68cc10f2e/kube-rbac-proxy/0.log" Jun 10 11:16:09.786415 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:09.786381 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/ovn-controller/0.log" Jun 10 11:16:09.807596 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:09.807568 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/ovn-acl-logging/0.log" Jun 10 11:16:09.849659 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:09.849622 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/kube-rbac-proxy-node/0.log" Jun 10 11:16:09.882076 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:09.882046 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/kube-rbac-proxy-ovn-metrics/0.log" Jun 10 11:16:09.901566 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:09.901540 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/northd/0.log" Jun 10 11:16:09.927461 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:09.927441 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/nbdb/0.log" Jun 10 11:16:09.952494 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:09.952470 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/sbdb/0.log" Jun 10 11:16:10.012323 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:10.012295 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pfkh8_91988be0-a23c-4f97-a30f-200ceb718da0/ovnkube-controller/0.log" Jun 10 11:16:11.110741 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:11.110669 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gdqfd_89fd8558-befa-433f-9575-27edb9ebdf62/network-check-target-container/0.log" Jun 10 11:16:12.128381 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:12.128348 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-498lq_6845d82a-8317-441b-b59a-41bc1b86c9df/iptables-alerter/0.log" Jun 10 11:16:13.004716 ip-10-0-128-191 kubenswrapper[2545]: I0610 11:16:13.004638 2545 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-p82p8_004c774c-b87b-4f93-a276-2cab75c42172/tuned/0.log"