Apr 21 17:33:16.164693 ip-10-0-143-230 systemd[1]: Starting Kubernetes Kubelet... Apr 21 17:33:16.594295 ip-10-0-143-230 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 21 17:33:16.594295 ip-10-0-143-230 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Apr 21 17:33:16.594295 ip-10-0-143-230 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 21 17:33:16.594295 ip-10-0-143-230 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Apr 21 17:33:16.594295 ip-10-0-143-230 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 21 17:33:16.596656 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.596571 2578 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Apr 21 17:33:16.603006 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.602991 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 21 17:33:16.603006 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603007 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603012 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603016 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603019 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603023 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603026 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603029 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603032 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603042 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603046 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603048 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603051 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603054 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603056 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603060 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603063 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603065 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603068 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603070 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603073 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 21 17:33:16.603076 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603075 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603078 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603080 2578 feature_gate.go:328] unrecognized feature gate: Example Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603083 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603086 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603089 2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603093 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603095 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603097 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603100 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603103 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603105 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603108 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603110 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603113 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603116 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603120 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603124 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603127 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603129 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 21 17:33:16.603565 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603132 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603134 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603137 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603139 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603142 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603144 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603147 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603149 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603151 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603154 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603156 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603159 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603161 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603164 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603169 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603171 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603174 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603177 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603180 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603182 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 21 17:33:16.604051 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603185 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603187 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603190 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603192 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603195 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603197 2578 feature_gate.go:328] unrecognized feature gate: Example2 Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603200 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603202 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603205 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603212 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603215 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603218 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603220 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603223 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603225 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603227 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603230 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603232 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603234 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603237 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 21 17:33:16.604567 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603239 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603242 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603244 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603247 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.603249 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604267 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604274 2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604278 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604281 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604284 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604286 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604289 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604291 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604294 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604298 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604300 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604303 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604306 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604309 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604311 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 21 17:33:16.605097 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604314 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604322 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604325 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604327 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604330 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604332 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604335 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604337 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604340 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604343 2578 feature_gate.go:328] unrecognized feature gate: Example Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604345 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604348 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604350 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604353 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604355 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604358 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604360 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604363 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604365 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604368 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 21 17:33:16.605604 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604370 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604372 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604375 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604378 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604380 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604383 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604386 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604389 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604391 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604396 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604400 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604403 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604406 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604424 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604435 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604438 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604440 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604443 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604445 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 21 17:33:16.606098 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604448 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604450 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604453 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604455 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604458 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604460 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604463 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604466 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604468 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604481 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604484 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604486 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604489 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604492 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604494 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604497 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604499 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604502 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604506 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 21 17:33:16.606575 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604509 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604511 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604514 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604518 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604520 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604523 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604525 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604528 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604530 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604539 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604542 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604544 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.604547 2578 feature_gate.go:328] unrecognized feature gate: Example2 Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604619 2578 flags.go:64] FLAG: --address="0.0.0.0" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604626 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604638 2578 flags.go:64] FLAG: --anonymous-auth="true" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604643 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604647 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604650 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604655 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604659 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604662 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Apr 21 17:33:16.607046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604665 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604668 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604672 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604675 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604677 2578 flags.go:64] FLAG: --cgroup-root="" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604680 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604683 2578 flags.go:64] FLAG: --client-ca-file="" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604686 2578 flags.go:64] FLAG: --cloud-config="" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604689 2578 flags.go:64] FLAG: --cloud-provider="external" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604692 2578 flags.go:64] FLAG: --cluster-dns="[]" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604700 2578 flags.go:64] FLAG: --cluster-domain="" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604703 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604706 2578 flags.go:64] FLAG: --config-dir="" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604709 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604713 2578 flags.go:64] FLAG: --container-log-max-files="5" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604716 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604719 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604722 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604726 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604728 2578 flags.go:64] FLAG: --contention-profiling="false" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604737 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604741 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604744 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604747 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604751 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Apr 21 17:33:16.607589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604754 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604757 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604760 2578 flags.go:64] FLAG: --enable-load-reader="false" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604763 2578 flags.go:64] FLAG: --enable-server="true" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604766 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604773 2578 flags.go:64] FLAG: --event-burst="100" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604776 2578 flags.go:64] FLAG: --event-qps="50" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604779 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604783 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604786 2578 flags.go:64] FLAG: --eviction-hard="" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604790 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604792 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604795 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604798 2578 flags.go:64] FLAG: --eviction-soft="" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604801 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604804 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604807 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604814 2578 flags.go:64] FLAG: --experimental-mounter-path="" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604817 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604820 2578 flags.go:64] FLAG: --fail-swap-on="true" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604823 2578 flags.go:64] FLAG: --feature-gates="" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604826 2578 flags.go:64] FLAG: --file-check-frequency="20s" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604829 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604833 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604836 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604839 2578 flags.go:64] FLAG: --healthz-port="10248" Apr 21 17:33:16.608198 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604842 2578 flags.go:64] FLAG: --help="false" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604845 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-230.ec2.internal" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604854 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604857 2578 flags.go:64] FLAG: --http-check-frequency="20s" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604860 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604863 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604866 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604869 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604872 2578 flags.go:64] FLAG: --image-service-endpoint="" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604875 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604878 2578 flags.go:64] FLAG: --kube-api-burst="100" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604880 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604883 2578 flags.go:64] FLAG: --kube-api-qps="50" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604886 2578 flags.go:64] FLAG: --kube-reserved="" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604889 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604892 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604895 2578 flags.go:64] FLAG: --kubelet-cgroups="" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604897 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604900 2578 flags.go:64] FLAG: --lock-file="" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604903 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604906 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604909 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604914 2578 flags.go:64] FLAG: --log-json-split-stream="false" Apr 21 17:33:16.608904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604918 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604921 2578 flags.go:64] FLAG: --log-text-split-stream="false" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604924 2578 flags.go:64] FLAG: --logging-format="text" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604927 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604930 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604933 2578 flags.go:64] FLAG: --manifest-url="" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604935 2578 flags.go:64] FLAG: --manifest-url-header="" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604940 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604943 2578 flags.go:64] FLAG: --max-open-files="1000000" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604947 2578 flags.go:64] FLAG: --max-pods="110" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604950 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604953 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604961 2578 flags.go:64] FLAG: --memory-manager-policy="None" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604964 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604967 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604970 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604973 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604981 2578 flags.go:64] FLAG: --node-status-max-images="50" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604984 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604987 2578 flags.go:64] FLAG: --oom-score-adj="-999" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604990 2578 flags.go:64] FLAG: --pod-cidr="" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604993 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8cfe89231412ff3ee8cb6207fa0be33cad0f08e88c9c0f1e9f7e8c6f14d6715" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.604999 2578 flags.go:64] FLAG: --pod-manifest-path="" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605002 2578 flags.go:64] FLAG: --pod-max-pids="-1" Apr 21 17:33:16.609486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605005 2578 flags.go:64] FLAG: --pods-per-core="0" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605008 2578 flags.go:64] FLAG: --port="10250" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605010 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605013 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d10da566d5b1bfaf" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605016 2578 flags.go:64] FLAG: --qos-reserved="" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605019 2578 flags.go:64] FLAG: --read-only-port="10255" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605022 2578 flags.go:64] FLAG: --register-node="true" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605025 2578 flags.go:64] FLAG: --register-schedulable="true" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605028 2578 flags.go:64] FLAG: --register-with-taints="" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605033 2578 flags.go:64] FLAG: --registry-burst="10" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605036 2578 flags.go:64] FLAG: --registry-qps="5" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605038 2578 flags.go:64] FLAG: --reserved-cpus="" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605041 2578 flags.go:64] FLAG: --reserved-memory="" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605045 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605048 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605051 2578 flags.go:64] FLAG: --rotate-certificates="false" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605054 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605057 2578 flags.go:64] FLAG: --runonce="false" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605060 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605063 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605066 2578 flags.go:64] FLAG: --seccomp-default="false" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605074 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605077 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605080 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605083 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605086 2578 flags.go:64] FLAG: --storage-driver-password="root" Apr 21 17:33:16.610095 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605089 2578 flags.go:64] FLAG: --storage-driver-secure="false" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605092 2578 flags.go:64] FLAG: --storage-driver-table="stats" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605095 2578 flags.go:64] FLAG: --storage-driver-user="root" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605097 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605100 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605104 2578 flags.go:64] FLAG: --system-cgroups="" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605106 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605112 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605114 2578 flags.go:64] FLAG: --tls-cert-file="" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605117 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605125 2578 flags.go:64] FLAG: --tls-min-version="" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605128 2578 flags.go:64] FLAG: --tls-private-key-file="" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605131 2578 flags.go:64] FLAG: --topology-manager-policy="none" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605134 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605137 2578 flags.go:64] FLAG: --topology-manager-scope="container" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605141 2578 flags.go:64] FLAG: --v="2" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605146 2578 flags.go:64] FLAG: --version="false" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605154 2578 flags.go:64] FLAG: --vmodule="" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605158 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.605161 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605282 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605286 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605290 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605294 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 21 17:33:16.610739 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605297 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605299 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605302 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605304 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605314 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605317 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605320 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605322 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605324 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605327 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605330 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605332 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605335 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605337 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605340 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605343 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605345 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605348 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605350 2578 feature_gate.go:328] unrecognized feature gate: Example2 Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605353 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 21 17:33:16.611307 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605356 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605358 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605360 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605364 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605366 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605369 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605372 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605374 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605378 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605382 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605384 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605389 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605392 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605394 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605397 2578 feature_gate.go:328] unrecognized feature gate: Example Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605401 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605423 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605432 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605435 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605438 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 21 17:33:16.611858 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605440 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605443 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605446 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605448 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605451 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605453 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605456 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605458 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605461 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605463 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605466 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605468 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605471 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605473 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605476 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605479 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605482 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605485 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605487 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605490 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 21 17:33:16.612433 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605492 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605495 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605498 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605505 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605507 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605510 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605513 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605515 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605518 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605520 2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605528 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605531 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605534 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605537 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605539 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605542 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605544 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605547 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605549 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605552 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 21 17:33:16.613017 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605554 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 21 17:33:16.613625 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.605557 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 21 17:33:16.613625 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.606572 2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 21 17:33:16.614887 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.614870 2578 server.go:530] "Kubelet version" kubeletVersion="v1.33.9" Apr 21 17:33:16.614928 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.614888 2578 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614933 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614939 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614942 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614946 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614949 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614952 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614954 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614957 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614960 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614963 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614966 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 21 17:33:16.614963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614968 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614971 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614974 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614977 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614979 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614982 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614984 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614987 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614990 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614992 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614995 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.614997 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615000 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615003 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615006 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615008 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615011 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615013 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615016 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615019 2578 feature_gate.go:328] unrecognized feature gate: Example2 Apr 21 17:33:16.615308 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615022 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615025 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615027 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615030 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615033 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615037 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615039 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615042 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615044 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615047 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615049 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615051 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615054 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615057 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615061 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615064 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615066 2578 feature_gate.go:328] unrecognized feature gate: Example Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615069 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615072 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 21 17:33:16.615872 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615074 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615077 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615081 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615085 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615088 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615091 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615093 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615096 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615099 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615101 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615104 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615106 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615109 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615111 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615115 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615117 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615120 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615122 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615125 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 21 17:33:16.616367 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615127 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615130 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615132 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615135 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615137 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615140 2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615143 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615145 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615148 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615151 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615154 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615156 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615159 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615162 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615164 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615167 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 21 17:33:16.616865 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615170 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.615175 2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615276 2578 feature_gate.go:328] unrecognized feature gate: Example2 Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615282 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615286 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615289 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615292 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615295 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615298 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615300 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615303 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615306 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615309 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615311 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615314 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 21 17:33:16.617259 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615317 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615321 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615325 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615328 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615331 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615334 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615337 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615341 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615344 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615347 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615350 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615353 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615355 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615358 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615361 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615363 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615366 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615368 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 21 17:33:16.617721 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615371 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615373 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615376 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615378 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615381 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615383 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615386 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615388 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615391 2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615393 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615396 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615398 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615401 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615403 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615406 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615408 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615421 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615424 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615426 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615429 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 21 17:33:16.618160 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615432 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615434 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615437 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615439 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615442 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615445 2578 feature_gate.go:328] unrecognized feature gate: Example Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615447 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615450 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615453 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615455 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615458 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615461 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615464 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615466 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615468 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615471 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615474 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615476 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615479 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615481 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 21 17:33:16.618684 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615484 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615486 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615489 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615491 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615494 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615496 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615499 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615502 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615504 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615506 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615509 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615512 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615514 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615517 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:16.615520 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.615525 2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 21 17:33:16.619181 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.616249 2578 server.go:962] "Client rotation is on, will bootstrap in background" Apr 21 17:33:16.619587 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.618163 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Apr 21 17:33:16.619587 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.619093 2578 server.go:1019] "Starting client certificate rotation" Apr 21 17:33:16.619587 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.619187 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 21 17:33:16.619587 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.619229 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 21 17:33:16.643196 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.643181 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 21 17:33:16.646511 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.646480 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 21 17:33:16.655247 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.655227 2578 log.go:25] "Validated CRI v1 runtime API" Apr 21 17:33:16.660709 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.660695 2578 log.go:25] "Validated CRI v1 image API" Apr 21 17:33:16.661827 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.661810 2578 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Apr 21 17:33:16.664053 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.664029 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90a7fdc0-3688-4c36-bea5-7adb602c880c:/dev/nvme0n1p4 acff0b46-a273-40f8-90bb-5f7545ed7a27:/dev/nvme0n1p3] Apr 21 17:33:16.664126 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.664053 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Apr 21 17:33:16.669719 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.669602 2578 manager.go:217] Machine: {Timestamp:2026-04-21 17:33:16.667869924 +0000 UTC m=+0.389984889 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100140 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec296fe14e72f89dab384dec8a0a5194 SystemUUID:ec296fe1-4e72-f89d-ab38-4dec8a0a5194 BootID:274b8ee3-2c99-4286-bbb5-f4fe0a119e96 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d9:a6:3a:3a:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d9:a6:3a:3a:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:db:79:7e:38:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 21 17:33:16.669719 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.669713 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Apr 21 17:33:16.669823 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.669813 2578 manager.go:233] Version: {KernelVersion:5.14.0-570.107.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260414-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Apr 21 17:33:16.670857 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.670832 2578 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Apr 21 17:33:16.670997 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.670858 2578 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Apr 21 17:33:16.671045 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.671007 2578 topology_manager.go:138] "Creating topology manager with none policy" Apr 21 17:33:16.671045 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.671016 2578 container_manager_linux.go:306] "Creating device plugin manager" Apr 21 17:33:16.671045 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.671033 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 21 17:33:16.671798 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.671782 2578 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 21 17:33:16.673020 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.673009 2578 state_mem.go:36] "Initialized new in-memory state store" Apr 21 17:33:16.673127 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.673118 2578 server.go:1267] "Using root directory" path="/var/lib/kubelet" Apr 21 17:33:16.674126 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.674108 2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 21 17:33:16.675233 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.675222 2578 kubelet.go:491] "Attempting to sync node with API server" Apr 21 17:33:16.675271 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.675236 2578 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Apr 21 17:33:16.675271 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.675248 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Apr 21 17:33:16.675271 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.675256 2578 kubelet.go:397] "Adding apiserver pod source" Apr 21 17:33:16.675271 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.675265 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Apr 21 17:33:16.676293 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.676280 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 21 17:33:16.676340 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.676298 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 21 17:33:16.679051 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.679035 2578 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1" Apr 21 17:33:16.680332 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.680319 2578 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Apr 21 17:33:16.682313 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682299 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682325 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682332 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682338 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682343 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682349 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682355 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682360 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Apr 21 17:33:16.682365 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682368 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Apr 21 17:33:16.682600 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682374 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Apr 21 17:33:16.682600 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682388 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Apr 21 17:33:16.682600 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.682397 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Apr 21 17:33:16.683280 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.683271 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Apr 21 17:33:16.683308 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.683281 2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Apr 21 17:33:16.687916 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.687892 2578 watchdog_linux.go:99] "Systemd watchdog is not enabled" Apr 21 17:33:16.688002 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.687943 2578 server.go:1295] "Started kubelet" Apr 21 17:33:16.688139 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.688092 2578 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Apr 21 17:33:16.688194 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.688171 2578 server_v1.go:47] "podresources" method="list" useActivePods=true Apr 21 17:33:16.688405 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.688379 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Apr 21 17:33:16.688568 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.688545 2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Apr 21 17:33:16.688641 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.688602 2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Apr 21 17:33:16.688741 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.688723 2578 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-230.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Apr 21 17:33:16.688806 ip-10-0-143-230 systemd[1]: Started Kubernetes Kubelet. Apr 21 17:33:16.690930 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.690914 2578 server.go:317] "Adding debug handlers to kubelet server" Apr 21 17:33:16.692033 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.692016 2578 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Apr 21 17:33:16.695619 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.695601 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Apr 21 17:33:16.696622 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.696595 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Apr 21 17:33:16.697188 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697170 2578 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Apr 21 17:33:16.697188 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697179 2578 volume_manager.go:295] "The desired_state_of_world populator starts" Apr 21 17:33:16.697307 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697203 2578 volume_manager.go:297] "Starting Kubelet Volume Manager" Apr 21 17:33:16.697307 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697222 2578 factory.go:55] Registering systemd factory Apr 21 17:33:16.697307 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697265 2578 factory.go:223] Registration of the systemd container factory successfully Apr 21 17:33:16.697307 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697294 2578 reconstruct.go:97] "Volume reconstruction finished" Apr 21 17:33:16.697307 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697306 2578 reconciler.go:26] "Reconciler: start to sync state" Apr 21 17:33:16.697534 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.697462 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:16.697534 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697524 2578 factory.go:153] Registering CRI-O factory Apr 21 17:33:16.697623 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697536 2578 factory.go:223] Registration of the crio container factory successfully Apr 21 17:33:16.697623 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697592 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Apr 21 17:33:16.697623 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697616 2578 factory.go:103] Registering Raw factory Apr 21 17:33:16.697744 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.697642 2578 manager.go:1196] Started watching for new ooms in manager Apr 21 17:33:16.698255 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.698243 2578 manager.go:319] Starting recovery of all containers Apr 21 17:33:16.698466 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.697517 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-230.ec2.internal.18a86fa27e0e36e4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-230.ec2.internal,UID:ip-10-0-143-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-230.ec2.internal,},FirstTimestamp:2026-04-21 17:33:16.687906532 +0000 UTC m=+0.410021497,LastTimestamp:2026-04-21 17:33:16.687906532 +0000 UTC m=+0.410021497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-230.ec2.internal,}" Apr 21 17:33:16.699198 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.699176 2578 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Apr 21 17:33:16.706113 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.705929 2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Apr 21 17:33:16.712022 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.712004 2578 manager.go:324] Recovery completed Apr 21 17:33:16.712128 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.711997 2578 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Apr 21 17:33:16.712530 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.712503 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Apr 21 17:33:16.713584 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.713563 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Apr 21 17:33:16.716770 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.716758 2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 21 17:33:16.719262 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.719244 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientMemory" Apr 21 17:33:16.719344 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.719278 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasNoDiskPressure" Apr 21 17:33:16.719344 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.719293 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientPID" Apr 21 17:33:16.719834 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.719820 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Apr 21 17:33:16.719884 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.719835 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Apr 21 17:33:16.719884 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.719860 2578 state_mem.go:36] "Initialized new in-memory state store" Apr 21 17:33:16.721614 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.721551 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-230.ec2.internal.18a86fa27feca6ba default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-230.ec2.internal,UID:ip-10-0-143-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-230.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-230.ec2.internal,},FirstTimestamp:2026-04-21 17:33:16.71926137 +0000 UTC m=+0.441376340,LastTimestamp:2026-04-21 17:33:16.71926137 +0000 UTC m=+0.441376340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-230.ec2.internal,}" Apr 21 17:33:16.722176 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.722129 2578 policy_none.go:49] "None policy: Start" Apr 21 17:33:16.722230 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.722181 2578 memory_manager.go:186] "Starting memorymanager" policy="None" Apr 21 17:33:16.722230 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.722193 2578 state_mem.go:35] "Initializing new in-memory state store" Apr 21 17:33:16.736102 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.736036 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-230.ec2.internal.18a86fa27fed0486 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-230.ec2.internal,UID:ip-10-0-143-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-230.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-230.ec2.internal,},FirstTimestamp:2026-04-21 17:33:16.719285382 +0000 UTC m=+0.441400351,LastTimestamp:2026-04-21 17:33:16.719285382 +0000 UTC m=+0.441400351,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-230.ec2.internal,}" Apr 21 17:33:16.748737 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.748674 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-230.ec2.internal.18a86fa27fed3916 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-230.ec2.internal,UID:ip-10-0-143-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-230.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-230.ec2.internal,},FirstTimestamp:2026-04-21 17:33:16.719298838 +0000 UTC m=+0.441413805,LastTimestamp:2026-04-21 17:33:16.719298838 +0000 UTC m=+0.441413805,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-230.ec2.internal,}" Apr 21 17:33:16.766127 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.766104 2578 manager.go:341] "Starting Device Plugin manager" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.766140 2578 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.766150 2578 server.go:85] "Starting device plugin registration server" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.766393 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.766407 2578 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.766521 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.766606 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.766616 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.767085 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Apr 21 17:33:16.768844 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.767127 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:16.779130 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.779066 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-230.ec2.internal.18a86fa282d919ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-230.ec2.internal,UID:ip-10-0-143-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-230.ec2.internal,},FirstTimestamp:2026-04-21 17:33:16.768311758 +0000 UTC m=+0.490426713,LastTimestamp:2026-04-21 17:33:16.768311758 +0000 UTC m=+0.490426713,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-230.ec2.internal,}" Apr 21 17:33:16.794214 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.794192 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nkxvq" Apr 21 17:33:16.796835 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.796818 2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Apr 21 17:33:16.796907 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.796843 2578 status_manager.go:230] "Starting to sync pod status with apiserver" Apr 21 17:33:16.796907 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.796859 2578 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Apr 21 17:33:16.796907 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.796865 2578 kubelet.go:2451] "Starting kubelet main sync loop" Apr 21 17:33:16.796907 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.796894 2578 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Apr 21 17:33:16.812706 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.812684 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nkxvq" Apr 21 17:33:16.827868 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.827850 2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 21 17:33:16.867313 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.867262 2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 21 17:33:16.869601 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.869586 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientMemory" Apr 21 17:33:16.869682 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.869615 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasNoDiskPressure" Apr 21 17:33:16.869682 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.869625 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientPID" Apr 21 17:33:16.869682 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.869646 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-230.ec2.internal" Apr 21 17:33:16.888387 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.888369 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-230.ec2.internal" Apr 21 17:33:16.888469 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.888389 2578 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-230.ec2.internal\": node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:16.897463 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.897443 2578 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal"] Apr 21 17:33:16.897520 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.897508 2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 21 17:33:16.898262 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.898249 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientMemory" Apr 21 17:33:16.898333 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.898274 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasNoDiskPressure" Apr 21 17:33:16.898333 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.898290 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientPID" Apr 21 17:33:16.899860 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.899848 2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 21 17:33:16.899986 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.899971 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:16.900061 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.900000 2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 21 17:33:16.900587 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.900563 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientMemory" Apr 21 17:33:16.900709 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.900594 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasNoDiskPressure" Apr 21 17:33:16.900709 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.900609 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientPID" Apr 21 17:33:16.900709 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.900567 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientMemory" Apr 21 17:33:16.900709 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.900641 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasNoDiskPressure" Apr 21 17:33:16.900709 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.900654 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientPID" Apr 21 17:33:16.901990 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.901975 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" Apr 21 17:33:16.902060 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.901999 2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 21 17:33:16.902584 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.902569 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientMemory" Apr 21 17:33:16.902681 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.902591 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasNoDiskPressure" Apr 21 17:33:16.902681 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:16.902631 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeHasSufficientPID" Apr 21 17:33:16.922645 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.922629 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:16.935093 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.935069 2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-230.ec2.internal\" not found" node="ip-10-0-143-230.ec2.internal" Apr 21 17:33:16.941333 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:16.941315 2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-230.ec2.internal\" not found" node="ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.023396 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.023372 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:17.098653 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.098623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/447b22b2a1623dee81a3ca7a095362cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal\" (UID: \"447b22b2a1623dee81a3ca7a095362cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.098653 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.098656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/447b22b2a1623dee81a3ca7a095362cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal\" (UID: \"447b22b2a1623dee81a3ca7a095362cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.098803 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.098677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e474269ec2989f5a8531df60ebc0fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-230.ec2.internal\" (UID: \"5e474269ec2989f5a8531df60ebc0fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.124264 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.124206 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:17.199533 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.199505 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/447b22b2a1623dee81a3ca7a095362cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal\" (UID: \"447b22b2a1623dee81a3ca7a095362cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.199651 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.199539 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/447b22b2a1623dee81a3ca7a095362cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal\" (UID: \"447b22b2a1623dee81a3ca7a095362cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.199651 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.199557 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e474269ec2989f5a8531df60ebc0fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-230.ec2.internal\" (UID: \"5e474269ec2989f5a8531df60ebc0fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.199651 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.199611 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/447b22b2a1623dee81a3ca7a095362cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal\" (UID: \"447b22b2a1623dee81a3ca7a095362cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.199651 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.199625 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/447b22b2a1623dee81a3ca7a095362cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal\" (UID: \"447b22b2a1623dee81a3ca7a095362cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.199776 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.199668 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5e474269ec2989f5a8531df60ebc0fd0-config\") pod \"kube-apiserver-proxy-ip-10-0-143-230.ec2.internal\" (UID: \"5e474269ec2989f5a8531df60ebc0fd0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.225304 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.225277 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:17.237481 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.237461 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.243915 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.243900 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.325746 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.325711 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:17.426232 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.426170 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:17.526706 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.526678 2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-230.ec2.internal\" not found" Apr 21 17:33:17.554794 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.554772 2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 21 17:33:17.597122 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.597085 2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.609581 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.609563 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 21 17:33:17.610551 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.610540 2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.618925 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.618906 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Apr 21 17:33:17.619070 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.619048 2578 kubelet.go:3342] "Failed creating a mirror pod" err="Post \"https://a589337ece5374fc0b8a4a38753524f1-41a9b3f72a0980c1.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.143.230:57022->44.193.241.172:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:17.619121 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.619068 2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 21 17:33:17.619121 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.619076 2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 21 17:33:17.676108 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.676082 2578 apiserver.go:52] "Watching apiserver" Apr 21 17:33:17.696538 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.696465 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Apr 21 17:33:17.697634 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.697616 2578 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Apr 21 17:33:17.697936 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.697918 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr","openshift-cluster-node-tuning-operator/tuned-bvgql","openshift-image-registry/node-ca-vcxwk","openshift-multus/multus-2z52h","openshift-network-operator/iptables-alerter-msdfg","kube-system/konnectivity-agent-vfwzr","kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal","openshift-multus/multus-additional-cni-plugins-9fvp6","openshift-multus/network-metrics-daemon-z46hp","openshift-network-diagnostics/network-check-target-5d95f","openshift-ovn-kubernetes/ovnkube-node-ms5gf"] Apr 21 17:33:17.700248 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.700226 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.701792 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.701760 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.702962 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.702939 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:17.703054 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.703011 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:17.704105 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.704091 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.705427 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.705395 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.706576 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.706559 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.706777 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.706685 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Apr 21 17:33:17.706777 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.706695 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.707431 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.707401 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Apr 21 17:33:17.708160 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.708148 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:17.709348 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.709335 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:17.709429 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.709387 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:17.710574 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.710560 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.710841 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.710825 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Apr 21 17:33:17.711091 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.711076 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Apr 21 17:33:17.713330 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.711574 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Apr 21 17:33:17.713330 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.711627 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Apr 21 17:33:17.713932 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.713776 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Apr 21 17:33:17.713932 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.713797 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Apr 21 17:33:17.713932 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.713838 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Apr 21 17:33:17.713932 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.713797 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Apr 21 17:33:17.713932 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.713895 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Apr 21 17:33:17.713932 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.713842 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hp4ht\"" Apr 21 17:33:17.714301 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714222 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Apr 21 17:33:17.714366 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714324 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n5nk4\"" Apr 21 17:33:17.714366 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714347 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Apr 21 17:33:17.714503 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714366 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shh2b\"" Apr 21 17:33:17.714503 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714445 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8xq45\"" Apr 21 17:33:17.714503 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714485 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Apr 21 17:33:17.714672 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714555 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Apr 21 17:33:17.714672 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714576 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Apr 21 17:33:17.714672 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714616 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Apr 21 17:33:17.714824 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714809 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Apr 21 17:33:17.714895 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714829 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jgxnc\"" Apr 21 17:33:17.714895 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714877 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Apr 21 17:33:17.715010 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.714981 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95lrc\"" Apr 21 17:33:17.715080 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715046 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g24gs\"" Apr 21 17:33:17.715080 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715056 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Apr 21 17:33:17.715187 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715110 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Apr 21 17:33:17.715187 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715058 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Apr 21 17:33:17.715187 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715135 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Apr 21 17:33:17.715187 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715182 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Apr 21 17:33:17.715493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715234 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Apr 21 17:33:17.715493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.715366 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q4hxb\"" Apr 21 17:33:17.722714 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.722693 2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 21 17:33:17.732130 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.732110 2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 21 17:33:17.746665 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.746645 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d427f" Apr 21 17:33:17.758555 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.758540 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d427f" Apr 21 17:33:17.777917 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:17.777884 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447b22b2a1623dee81a3ca7a095362cd.slice/crio-0d5be3589214adf6ab4adbd1fb58b8af29e77084c787ce0749c7284c111ee588 WatchSource:0}: Error finding container 0d5be3589214adf6ab4adbd1fb58b8af29e77084c787ce0749c7284c111ee588: Status 404 returned error can't find the container with id 0d5be3589214adf6ab4adbd1fb58b8af29e77084c787ce0749c7284c111ee588 Apr 21 17:33:17.778179 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:17.778158 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e474269ec2989f5a8531df60ebc0fd0.slice/crio-0b4fd5d0833bbaf58ca5686d106cdc25b12b5fed0a319b0c5c025b2b25350e64 WatchSource:0}: Error finding container 0b4fd5d0833bbaf58ca5686d106cdc25b12b5fed0a319b0c5c025b2b25350e64: Status 404 returned error can't find the container with id 0b4fd5d0833bbaf58ca5686d106cdc25b12b5fed0a319b0c5c025b2b25350e64 Apr 21 17:33:17.782169 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.782154 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 21 17:33:17.798373 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.798358 2578 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Apr 21 17:33:17.799352 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.799318 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" event={"ID":"447b22b2a1623dee81a3ca7a095362cd","Type":"ContainerStarted","Data":"0d5be3589214adf6ab4adbd1fb58b8af29e77084c787ce0749c7284c111ee588"} Apr 21 17:33:17.800468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.800426 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" event={"ID":"5e474269ec2989f5a8531df60ebc0fd0","Type":"ContainerStarted","Data":"0b4fd5d0833bbaf58ca5686d106cdc25b12b5fed0a319b0c5c025b2b25350e64"} Apr 21 17:33:17.803037 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-var-lib-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.803125 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwf6\" (UniqueName: \"kubernetes.io/projected/547c1477-65a3-4469-8fb5-5700ad4cd216-kube-api-access-qmwf6\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.803125 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-hostroot\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.803125 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-os-release\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.803125 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-device-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803166 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-kube-api-access-lkg8s\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnk2j\" (UniqueName: \"kubernetes.io/projected/1f3425f9-f78c-4583-9efa-71d0ba28a448-kube-api-access-jnk2j\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-kubelet\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-cni-bin\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovn-node-metrics-cert\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.803286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b36fb126-3aba-45e0-8d63-ad3846dcc93a-konnectivity-ca\") pod \"konnectivity-agent-vfwzr\" (UID: \"b36fb126-3aba-45e0-8d63-ad3846dcc93a\") " pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-system-cni-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-registration-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-os-release\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-multus-certs\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm2lg\" (UniqueName: \"kubernetes.io/projected/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-kube-api-access-gm2lg\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-socket-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-systemd\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-tuned\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-systemd\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.803680 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-cni-bin\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-cni-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3521509-adc6-48b2-905b-f7597ce17704-multus-daemon-config\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-kubernetes\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-sys\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-k8s-cni-cncf-io\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-etc-kubernetes\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69d2\" (UniqueName: \"kubernetes.io/projected/b3521509-adc6-48b2-905b-f7597ce17704-kube-api-access-p69d2\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.803981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ql8\" (UniqueName: \"kubernetes.io/projected/cbb61ff1-0bdf-4220-99ee-6f860546dc41-kube-api-access-s4ql8\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-host\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38857837-ebee-4764-9e74-c45e932e9f87-iptables-alerter-script\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-node-log\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-cni-netd\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovnkube-script-lib\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-kubelet\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysctl-d\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-var-lib-kubelet\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38857837-ebee-4764-9e74-c45e932e9f87-host-slash\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-env-overrides\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-ovn\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-conf-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-etc-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-system-cni-dir\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysctl-conf\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-log-socket\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovnkube-config\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/547c1477-65a3-4469-8fb5-5700ad4cd216-host\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.804752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-cni-multus\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-run\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/547c1477-65a3-4469-8fb5-5700ad4cd216-serviceca\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-modprobe-d\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-slash\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfxlk\" (UniqueName: \"kubernetes.io/projected/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-kube-api-access-kfxlk\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-cnibin\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-netns\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdnhw\" (UniqueName: \"kubernetes.io/projected/38857837-ebee-4764-9e74-c45e932e9f87-kube-api-access-xdnhw\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-socket-dir-parent\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-lib-modules\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b36fb126-3aba-45e0-8d63-ad3846dcc93a-agent-certs\") pod \"konnectivity-agent-vfwzr\" (UID: \"b36fb126-3aba-45e0-8d63-ad3846dcc93a\") " pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804782 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-sys-fs\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysconfig\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804884 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-systemd-units\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3521509-adc6-48b2-905b-f7597ce17704-cni-binary-copy\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.805291 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cnibin\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.805770 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f3425f9-f78c-4583-9efa-71d0ba28a448-tmp\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.805770 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.804987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-run-netns\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.815399 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.815368 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-20 17:28:16 +0000 UTC" deadline="2028-01-10 08:37:20.405180644 +0000 UTC" Apr 21 17:33:17.815399 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.815391 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15087h4m2.589791373s" Apr 21 17:33:17.905207 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905169 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p69d2\" (UniqueName: \"kubernetes.io/projected/b3521509-adc6-48b2-905b-f7597ce17704-kube-api-access-p69d2\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.905207 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905208 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.905444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905225 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ql8\" (UniqueName: \"kubernetes.io/projected/cbb61ff1-0bdf-4220-99ee-6f860546dc41-kube-api-access-s4ql8\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:17.905444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905240 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-host\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.905444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905265 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38857837-ebee-4764-9e74-c45e932e9f87-iptables-alerter-script\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.905444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905287 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-node-log\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905328 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-node-log\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905345 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-host\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.905444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905409 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-cni-netd\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-cni-netd\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905470 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovnkube-script-lib\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905497 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-kubelet\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905523 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysctl-d\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905552 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-var-lib-kubelet\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905556 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-kubelet\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905577 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38857837-ebee-4764-9e74-c45e932e9f87-host-slash\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-var-lib-kubelet\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905602 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-env-overrides\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38857837-ebee-4764-9e74-c45e932e9f87-host-slash\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905652 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-ovn\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905677 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905702 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-conf-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905717 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysctl-d\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905727 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-etc-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.905752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905752 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-system-cni-dir\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905777 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905796 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905802 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysctl-conf\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905839 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-log-socket\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905862 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovnkube-config\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905884 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/547c1477-65a3-4469-8fb5-5700ad4cd216-host\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905893 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905909 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-cni-multus\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysctl-conf\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905893 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38857837-ebee-4764-9e74-c45e932e9f87-iptables-alerter-script\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905935 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-run\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905958 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/547c1477-65a3-4469-8fb5-5700ad4cd216-serviceca\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-ovn\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905982 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-modprobe-d\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-system-cni-dir\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.905977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-etc-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.906493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906018 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-slash\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-log-socket\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906045 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kfxlk\" (UniqueName: \"kubernetes.io/projected/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-kube-api-access-kfxlk\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/547c1477-65a3-4469-8fb5-5700ad4cd216-host\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906069 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-cnibin\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906092 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-netns\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906116 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdnhw\" (UniqueName: \"kubernetes.io/projected/38857837-ebee-4764-9e74-c45e932e9f87-kube-api-access-xdnhw\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906140 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-socket-dir-parent\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-modprobe-d\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906165 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-lib-modules\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-env-overrides\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906191 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b36fb126-3aba-45e0-8d63-ad3846dcc93a-agent-certs\") pod \"konnectivity-agent-vfwzr\" (UID: \"b36fb126-3aba-45e0-8d63-ad3846dcc93a\") " pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906171 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovnkube-script-lib\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906220 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-conf-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906244 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906242 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-netns\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906278 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-sys-fs\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906320 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysconfig\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.907326 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906345 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-systemd-units\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906369 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3521509-adc6-48b2-905b-f7597ce17704-cni-binary-copy\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-slash\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906395 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cnibin\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906438 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-lib-modules\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906444 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f3425f9-f78c-4583-9efa-71d0ba28a448-tmp\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovnkube-config\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-sys-fs\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/547c1477-65a3-4469-8fb5-5700ad4cd216-serviceca\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-cni-multus\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906513 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cnibin\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-socket-dir-parent\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-cnibin\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906619 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-run-netns\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906665 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-var-lib-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906658 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-run-netns\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906696 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwf6\" (UniqueName: \"kubernetes.io/projected/547c1477-65a3-4469-8fb5-5700ad4cd216-kube-api-access-qmwf6\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906750 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-hostroot\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.908736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-sysconfig\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906717 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-var-lib-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906777 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-os-release\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-run\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906810 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-device-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-hostroot\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906812 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906834 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906852 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-os-release\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906866 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-kube-api-access-lkg8s\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906884 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-device-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906938 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jnk2j\" (UniqueName: \"kubernetes.io/projected/1f3425f9-f78c-4583-9efa-71d0ba28a448-kube-api-access-jnk2j\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906935 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906968 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-kubelet\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907012 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-cni-bin\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907043 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.906580 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.909586 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907074 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3521509-adc6-48b2-905b-f7597ce17704-cni-binary-copy\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907087 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovn-node-metrics-cert\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-systemd-units\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907116 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b36fb126-3aba-45e0-8d63-ad3846dcc93a-konnectivity-ca\") pod \"konnectivity-agent-vfwzr\" (UID: \"b36fb126-3aba-45e0-8d63-ad3846dcc93a\") " pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-var-lib-cni-bin\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907140 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-system-cni-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907182 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-registration-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907218 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907260 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-os-release\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-system-cni-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-kubelet\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907289 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-registration-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907285 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-multus-certs\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907333 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-openvswitch\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907335 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907368 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-os-release\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907372 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gm2lg\" (UniqueName: \"kubernetes.io/projected/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-kube-api-access-gm2lg\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.910153 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-multus-certs\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907406 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907452 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-socket-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907476 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-systemd\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907498 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-tuned\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907522 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-systemd\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907545 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-cni-bin\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907570 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-cni-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907596 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-systemd\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907592 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3521509-adc6-48b2-905b-f7597ce17704-multus-daemon-config\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-socket-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-run-systemd\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.907661 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907670 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-multus-cni-dir\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-cni-bin\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.907754 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:18.407705804 +0000 UTC m=+2.129820763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:17.910641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907781 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907808 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-kubernetes\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907831 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-sys\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907842 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907857 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907885 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-k8s-cni-cncf-io\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-kubernetes\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907914 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-etc-kubernetes\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b36fb126-3aba-45e0-8d63-ad3846dcc93a-konnectivity-ca\") pod \"konnectivity-agent-vfwzr\" (UID: \"b36fb126-3aba-45e0-8d63-ad3846dcc93a\") " pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.907993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3425f9-f78c-4583-9efa-71d0ba28a448-sys\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.908026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-host-run-k8s-cni-cncf-io\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.908039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3521509-adc6-48b2-905b-f7597ce17704-etc-kubernetes\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.908077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3521509-adc6-48b2-905b-f7597ce17704-multus-daemon-config\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.908375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.909874 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f3425f9-f78c-4583-9efa-71d0ba28a448-tmp\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.909948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-ovn-node-metrics-cert\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.910030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f3425f9-f78c-4583-9efa-71d0ba28a448-etc-tuned\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.911094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.910113 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b36fb126-3aba-45e0-8d63-ad3846dcc93a-agent-certs\") pod \"konnectivity-agent-vfwzr\" (UID: \"b36fb126-3aba-45e0-8d63-ad3846dcc93a\") " pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:17.913564 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.913543 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 21 17:33:17.913564 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.913560 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 21 17:33:17.913564 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.913569 2578 projected.go:194] Error preparing data for projected volume kube-api-access-54hvt for pod openshift-network-diagnostics/network-check-target-5d95f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:17.913708 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:17.913609 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt podName:9959f536-1d43-45dc-a1b9-a84e76dbafa0 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:18.413597585 +0000 UTC m=+2.135712559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-54hvt" (UniqueName: "kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt") pod "network-check-target-5d95f" (UID: "9959f536-1d43-45dc-a1b9-a84e76dbafa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:17.913708 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.913639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ql8\" (UniqueName: \"kubernetes.io/projected/cbb61ff1-0bdf-4220-99ee-6f860546dc41-kube-api-access-s4ql8\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:17.915702 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.915686 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwf6\" (UniqueName: \"kubernetes.io/projected/547c1477-65a3-4469-8fb5-5700ad4cd216-kube-api-access-qmwf6\") pod \"node-ca-vcxwk\" (UID: \"547c1477-65a3-4469-8fb5-5700ad4cd216\") " pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:17.916966 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.916951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfxlk\" (UniqueName: \"kubernetes.io/projected/abdbee8d-8443-4ccd-a3d5-ef918b3fc39a-kube-api-access-kfxlk\") pod \"ovnkube-node-ms5gf\" (UID: \"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:17.917972 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.917954 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69d2\" (UniqueName: \"kubernetes.io/projected/b3521509-adc6-48b2-905b-f7597ce17704-kube-api-access-p69d2\") pod \"multus-2z52h\" (UID: \"b3521509-adc6-48b2-905b-f7597ce17704\") " pod="openshift-multus/multus-2z52h" Apr 21 17:33:17.920064 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.920038 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkg8s\" (UniqueName: \"kubernetes.io/projected/6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3-kube-api-access-lkg8s\") pod \"aws-ebs-csi-driver-node-bvdpr\" (UID: \"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:17.920221 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.920191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm2lg\" (UniqueName: \"kubernetes.io/projected/c70a5efa-9e1c-4b04-b2df-fa59050fcd7e-kube-api-access-gm2lg\") pod \"multus-additional-cni-plugins-9fvp6\" (UID: \"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e\") " pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:17.920397 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.920378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnk2j\" (UniqueName: \"kubernetes.io/projected/1f3425f9-f78c-4583-9efa-71d0ba28a448-kube-api-access-jnk2j\") pod \"tuned-bvgql\" (UID: \"1f3425f9-f78c-4583-9efa-71d0ba28a448\") " pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:17.920614 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:17.920600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdnhw\" (UniqueName: \"kubernetes.io/projected/38857837-ebee-4764-9e74-c45e932e9f87-kube-api-access-xdnhw\") pod \"iptables-alerter-msdfg\" (UID: \"38857837-ebee-4764-9e74-c45e932e9f87\") " pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:18.022799 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.022742 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2z52h" Apr 21 17:33:18.029363 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:18.029337 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3521509_adc6_48b2_905b_f7597ce17704.slice/crio-7a07d6d2a3c388442ff365707b5e6f405b3d4239e076a9fb23e91641cc8fb7ac WatchSource:0}: Error finding container 7a07d6d2a3c388442ff365707b5e6f405b3d4239e076a9fb23e91641cc8fb7ac: Status 404 returned error can't find the container with id 7a07d6d2a3c388442ff365707b5e6f405b3d4239e076a9fb23e91641cc8fb7ac Apr 21 17:33:18.038112 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.038096 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" Apr 21 17:33:18.043606 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:18.043587 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70a5efa_9e1c_4b04_b2df_fa59050fcd7e.slice/crio-a8dd07775cdf5e720fc2e325ac5e7e0f9ad8da680709561aed53c610a05cbe76 WatchSource:0}: Error finding container a8dd07775cdf5e720fc2e325ac5e7e0f9ad8da680709561aed53c610a05cbe76: Status 404 returned error can't find the container with id a8dd07775cdf5e720fc2e325ac5e7e0f9ad8da680709561aed53c610a05cbe76 Apr 21 17:33:18.061178 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.061158 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" Apr 21 17:33:18.068380 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:18.068360 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6555c8ac_e4fb_4eb6_a7f7_c39b8a7c2fd3.slice/crio-8ae02b51c864db5de0702312c16fc92a268fe175a6d700fb8682d2207f04bc91 WatchSource:0}: Error finding container 8ae02b51c864db5de0702312c16fc92a268fe175a6d700fb8682d2207f04bc91: Status 404 returned error can't find the container with id 8ae02b51c864db5de0702312c16fc92a268fe175a6d700fb8682d2207f04bc91 Apr 21 17:33:18.069687 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.069667 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bvgql" Apr 21 17:33:18.087252 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.087231 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vcxwk" Apr 21 17:33:18.092427 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.092391 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msdfg" Apr 21 17:33:18.092706 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:18.092682 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547c1477_65a3_4469_8fb5_5700ad4cd216.slice/crio-c1822481aedb182e7e30e3c92f5aafc27e51a246359bba560a747f76e2333383 WatchSource:0}: Error finding container c1822481aedb182e7e30e3c92f5aafc27e51a246359bba560a747f76e2333383: Status 404 returned error can't find the container with id c1822481aedb182e7e30e3c92f5aafc27e51a246359bba560a747f76e2333383 Apr 21 17:33:18.097715 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:18.097693 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38857837_ebee_4764_9e74_c45e932e9f87.slice/crio-d71507ce666be7b27140315a25e5486c3248d82ae710a71726e464a34e5e289a WatchSource:0}: Error finding container d71507ce666be7b27140315a25e5486c3248d82ae710a71726e464a34e5e289a: Status 404 returned error can't find the container with id d71507ce666be7b27140315a25e5486c3248d82ae710a71726e464a34e5e289a Apr 21 17:33:18.098454 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.098440 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:18.103677 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.103661 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:18.103905 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:18.103884 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36fb126_3aba_45e0_8d63_ad3846dcc93a.slice/crio-fc643b8b203e0cb40f40c89e9253b6e620cc5baba029d6dbf8d868c711d8c48f WatchSource:0}: Error finding container fc643b8b203e0cb40f40c89e9253b6e620cc5baba029d6dbf8d868c711d8c48f: Status 404 returned error can't find the container with id fc643b8b203e0cb40f40c89e9253b6e620cc5baba029d6dbf8d868c711d8c48f Apr 21 17:33:18.109355 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:18.109337 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdbee8d_8443_4ccd_a3d5_ef918b3fc39a.slice/crio-db789c819d19ad579a8f83fadbe63fa5800f0076933f7c8335008d1a83633c00 WatchSource:0}: Error finding container db789c819d19ad579a8f83fadbe63fa5800f0076933f7c8335008d1a83633c00: Status 404 returned error can't find the container with id db789c819d19ad579a8f83fadbe63fa5800f0076933f7c8335008d1a83633c00 Apr 21 17:33:18.238655 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.238629 2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 21 17:33:18.411538 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.411451 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:18.411687 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:18.411615 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:18.411687 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:18.411681 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:19.411661299 +0000 UTC m=+3.133776259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:18.512400 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.512362 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:18.512601 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:18.512576 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 21 17:33:18.512601 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:18.512600 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 21 17:33:18.512704 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:18.512612 2578 projected.go:194] Error preparing data for projected volume kube-api-access-54hvt for pod openshift-network-diagnostics/network-check-target-5d95f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:18.512704 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:18.512681 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt podName:9959f536-1d43-45dc-a1b9-a84e76dbafa0 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:19.512662619 +0000 UTC m=+3.234777576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-54hvt" (UniqueName: "kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt") pod "network-check-target-5d95f" (UID: "9959f536-1d43-45dc-a1b9-a84e76dbafa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:18.759694 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.759614 2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 21 17:33:18.760366 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.760022 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-20 17:28:17 +0000 UTC" deadline="2027-10-21 09:45:03.724719547 +0000 UTC" Apr 21 17:33:18.760366 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.760043 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13144h11m44.964679165s" Apr 21 17:33:18.830373 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.830327 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerStarted","Data":"a8dd07775cdf5e720fc2e325ac5e7e0f9ad8da680709561aed53c610a05cbe76"} Apr 21 17:33:18.833255 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.833223 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vfwzr" event={"ID":"b36fb126-3aba-45e0-8d63-ad3846dcc93a","Type":"ContainerStarted","Data":"fc643b8b203e0cb40f40c89e9253b6e620cc5baba029d6dbf8d868c711d8c48f"} Apr 21 17:33:18.845105 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.845046 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msdfg" event={"ID":"38857837-ebee-4764-9e74-c45e932e9f87","Type":"ContainerStarted","Data":"d71507ce666be7b27140315a25e5486c3248d82ae710a71726e464a34e5e289a"} Apr 21 17:33:18.862540 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.862511 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vcxwk" event={"ID":"547c1477-65a3-4469-8fb5-5700ad4cd216","Type":"ContainerStarted","Data":"c1822481aedb182e7e30e3c92f5aafc27e51a246359bba560a747f76e2333383"} Apr 21 17:33:18.884017 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.883639 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2z52h" event={"ID":"b3521509-adc6-48b2-905b-f7597ce17704","Type":"ContainerStarted","Data":"7a07d6d2a3c388442ff365707b5e6f405b3d4239e076a9fb23e91641cc8fb7ac"} Apr 21 17:33:18.895639 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.895604 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"db789c819d19ad579a8f83fadbe63fa5800f0076933f7c8335008d1a83633c00"} Apr 21 17:33:18.913669 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.913643 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvgql" event={"ID":"1f3425f9-f78c-4583-9efa-71d0ba28a448","Type":"ContainerStarted","Data":"9a9d3bc5159c466b4c585e0563594456d09b095b85b086067768c65299931499"} Apr 21 17:33:18.927791 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:18.927759 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" event={"ID":"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3","Type":"ContainerStarted","Data":"8ae02b51c864db5de0702312c16fc92a268fe175a6d700fb8682d2207f04bc91"} Apr 21 17:33:19.419518 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:19.419481 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:19.419711 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.419676 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:19.419773 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.419737 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:21.419719573 +0000 UTC m=+5.141834530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:19.521305 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:19.520688 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:19.521305 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.520841 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 21 17:33:19.521305 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.520860 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 21 17:33:19.521305 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.520873 2578 projected.go:194] Error preparing data for projected volume kube-api-access-54hvt for pod openshift-network-diagnostics/network-check-target-5d95f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:19.521305 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.520928 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt podName:9959f536-1d43-45dc-a1b9-a84e76dbafa0 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:21.520908753 +0000 UTC m=+5.243023728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-54hvt" (UniqueName: "kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt") pod "network-check-target-5d95f" (UID: "9959f536-1d43-45dc-a1b9-a84e76dbafa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:19.761250 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:19.761164 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-20 17:28:17 +0000 UTC" deadline="2028-01-29 07:00:02.252500466 +0000 UTC" Apr 21 17:33:19.761250 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:19.761202 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15541h26m42.491301959s" Apr 21 17:33:19.797728 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:19.797692 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:19.797904 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.797828 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:19.798261 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:19.798240 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:19.798359 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:19.798332 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:19.834724 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:19.834696 2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 21 17:33:21.438533 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:21.438493 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:21.438994 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.438658 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:21.438994 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.438739 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:25.438719429 +0000 UTC m=+9.160834397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:21.539156 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:21.539110 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:21.539327 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.539306 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 21 17:33:21.539435 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.539328 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 21 17:33:21.539435 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.539353 2578 projected.go:194] Error preparing data for projected volume kube-api-access-54hvt for pod openshift-network-diagnostics/network-check-target-5d95f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:21.539435 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.539429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt podName:9959f536-1d43-45dc-a1b9-a84e76dbafa0 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:25.539395054 +0000 UTC m=+9.261510007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-54hvt" (UniqueName: "kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt") pod "network-check-target-5d95f" (UID: "9959f536-1d43-45dc-a1b9-a84e76dbafa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:21.797270 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:21.797171 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:21.797459 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:21.797204 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:21.797459 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.797346 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:21.797790 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:21.797767 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:23.798011 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:23.797963 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:23.798468 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:23.798101 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:23.798468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:23.798291 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:23.798468 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:23.798382 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:25.468456 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:25.468403 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:25.468923 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.468499 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:25.468923 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.468573 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:33.468554863 +0000 UTC m=+17.190669830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:25.569364 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:25.569325 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:25.569556 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.569531 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 21 17:33:25.569556 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.569556 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 21 17:33:25.569667 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.569569 2578 projected.go:194] Error preparing data for projected volume kube-api-access-54hvt for pod openshift-network-diagnostics/network-check-target-5d95f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:25.569667 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.569625 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt podName:9959f536-1d43-45dc-a1b9-a84e76dbafa0 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:33.569605517 +0000 UTC m=+17.291720475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-54hvt" (UniqueName: "kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt") pod "network-check-target-5d95f" (UID: "9959f536-1d43-45dc-a1b9-a84e76dbafa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:25.797345 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:25.797311 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:25.797531 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.797475 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:25.797599 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:25.797541 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:25.797665 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:25.797633 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:27.797571 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:27.797480 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:27.798019 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:27.797479 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:27.798019 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:27.797630 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:27.798019 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:27.797732 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:29.797567 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:29.797528 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:29.798005 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:29.797538 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:29.798005 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:29.797653 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:29.798005 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:29.797736 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:31.797555 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:31.797510 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:31.797962 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:31.797627 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:31.797962 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:31.797685 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:31.797962 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:31.797794 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:33.524423 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:33.524195 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:33.524918 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.524340 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:33.524918 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.524524 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:49.52450393 +0000 UTC m=+33.246618883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 21 17:33:33.625527 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:33.625489 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:33.625701 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.625647 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 21 17:33:33.625701 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.625668 2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 21 17:33:33.625701 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.625681 2578 projected.go:194] Error preparing data for projected volume kube-api-access-54hvt for pod openshift-network-diagnostics/network-check-target-5d95f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:33.625816 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.625735 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt podName:9959f536-1d43-45dc-a1b9-a84e76dbafa0 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:49.625719008 +0000 UTC m=+33.347833960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-54hvt" (UniqueName: "kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt") pod "network-check-target-5d95f" (UID: "9959f536-1d43-45dc-a1b9-a84e76dbafa0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 21 17:33:33.797114 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:33.797032 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:33.797264 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:33.797040 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:33.797264 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.797151 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:33.797377 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:33.797256 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:34.002676 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.002643 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8dcr9"] Apr 21 17:33:34.017024 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.016999 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.017183 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:34.017082 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:34.130196 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.130160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcf8f14a-09d1-4071-94df-ba4064d021a4-dbus\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.130368 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.130217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcf8f14a-09d1-4071-94df-ba4064d021a4-kubelet-config\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.130368 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.130244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.230761 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.230724 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcf8f14a-09d1-4071-94df-ba4064d021a4-dbus\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.230928 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.230769 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcf8f14a-09d1-4071-94df-ba4064d021a4-kubelet-config\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.230928 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.230799 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.230928 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.230889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcf8f14a-09d1-4071-94df-ba4064d021a4-kubelet-config\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.231077 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:34.230897 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:34.231077 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.230971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcf8f14a-09d1-4071-94df-ba4064d021a4-dbus\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.231077 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:34.230994 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret podName:dcf8f14a-09d1-4071-94df-ba4064d021a4 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:34.730973212 +0000 UTC m=+18.453088170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret") pod "global-pull-secret-syncer-8dcr9" (UID: "dcf8f14a-09d1-4071-94df-ba4064d021a4") : object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:34.274894 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.274863 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5n859"] Apr 21 17:33:34.296607 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.296582 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.316598 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.316572 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nzsmm\"" Apr 21 17:33:34.324961 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.324940 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Apr 21 17:33:34.325109 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.324943 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Apr 21 17:33:34.432672 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.432593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951bbb2d-337a-4723-9e9c-ef08f471f1fa-tmp-dir\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.432672 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.432654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/951bbb2d-337a-4723-9e9c-ef08f471f1fa-hosts-file\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.432896 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.432683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lg8\" (UniqueName: \"kubernetes.io/projected/951bbb2d-337a-4723-9e9c-ef08f471f1fa-kube-api-access-47lg8\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.534052 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.534016 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951bbb2d-337a-4723-9e9c-ef08f471f1fa-tmp-dir\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.534549 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.534071 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/951bbb2d-337a-4723-9e9c-ef08f471f1fa-hosts-file\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.534549 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.534101 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-47lg8\" (UniqueName: \"kubernetes.io/projected/951bbb2d-337a-4723-9e9c-ef08f471f1fa-kube-api-access-47lg8\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.534549 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.534216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/951bbb2d-337a-4723-9e9c-ef08f471f1fa-hosts-file\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.534549 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.534406 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/951bbb2d-337a-4723-9e9c-ef08f471f1fa-tmp-dir\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.544649 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.544623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lg8\" (UniqueName: \"kubernetes.io/projected/951bbb2d-337a-4723-9e9c-ef08f471f1fa-kube-api-access-47lg8\") pod \"node-resolver-5n859\" (UID: \"951bbb2d-337a-4723-9e9c-ef08f471f1fa\") " pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.606499 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.606466 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n859" Apr 21 17:33:34.736170 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:34.736056 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:34.736312 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:34.736225 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:34.736312 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:34.736287 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret podName:dcf8f14a-09d1-4071-94df-ba4064d021a4 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:35.736271564 +0000 UTC m=+19.458386516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret") pod "global-pull-secret-syncer-8dcr9" (UID: "dcf8f14a-09d1-4071-94df-ba4064d021a4") : object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:35.746823 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.746787 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:35.747167 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:35.746913 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:35.747167 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:35.746984 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret podName:dcf8f14a-09d1-4071-94df-ba4064d021a4 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:37.746964562 +0000 UTC m=+21.469079527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret") pod "global-pull-secret-syncer-8dcr9" (UID: "dcf8f14a-09d1-4071-94df-ba4064d021a4") : object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:35.797485 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.797344 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:35.797594 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.797449 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:35.797594 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:35.797564 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:35.797716 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.797456 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:35.797716 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:35.797629 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:35.797819 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:35.797722 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:35.980061 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.980033 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2z52h" event={"ID":"b3521509-adc6-48b2-905b-f7597ce17704","Type":"ContainerStarted","Data":"24a8292c06553bcb606f6eac5ac3fdf638ff832763070b76845147d85d9f1e95"} Apr 21 17:33:35.981293 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.981267 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n859" event={"ID":"951bbb2d-337a-4723-9e9c-ef08f471f1fa","Type":"ContainerStarted","Data":"58febd89ad9278173ae0ec63f8ca623eb64f7088119154e719d7e31059c7f6d2"} Apr 21 17:33:35.996281 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.995922 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:33:35.999934 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.999902 2578 generic.go:358] "Generic (PLEG): container finished" podID="abdbee8d-8443-4ccd-a3d5-ef918b3fc39a" containerID="35fe362212d85b9c3a0be6644d8e7e4714c35cea6ccf1a37aec5944320df73d1" exitCode=1 Apr 21 17:33:36.000039 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:35.999990 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"79c50ae1dd6f6f7db255221c9b2b0633e537e6a18613f0932e4697ee0c8ae087"} Apr 21 17:33:36.000039 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.000025 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"8ac694d2859d345ce0157198be87017b03eab8b5686da601223e48578ade3f0e"} Apr 21 17:33:36.000136 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.000040 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"3393333b7749b09609dbd40d3da6194556b0a54e5de355c4128fdd12c63d6fcf"} Apr 21 17:33:36.000136 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.000054 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerDied","Data":"35fe362212d85b9c3a0be6644d8e7e4714c35cea6ccf1a37aec5944320df73d1"} Apr 21 17:33:36.000136 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.000069 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"94a09d12872ec1fad106ec226399d685e5b7a518d9037ea656803f46322d83e2"} Apr 21 17:33:36.003519 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.003465 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bvgql" event={"ID":"1f3425f9-f78c-4583-9efa-71d0ba28a448","Type":"ContainerStarted","Data":"f35dc9bb6f73b341e20af033f2708490c70269ae2dc946686ed011ffe0f973b7"} Apr 21 17:33:36.005908 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.005879 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" event={"ID":"5e474269ec2989f5a8531df60ebc0fd0","Type":"ContainerStarted","Data":"59841b7c7b95fc44235d9598d2d573a7e280fafdcbd9666420b1e57abc71c19e"} Apr 21 17:33:36.032678 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.032614 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-2z52h" podStartSLOduration=2.384969021 podStartE2EDuration="20.032596103s" podCreationTimestamp="2026-04-21 17:33:16 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.030930863 +0000 UTC m=+1.753045816" lastFinishedPulling="2026-04-21 17:33:35.678557938 +0000 UTC m=+19.400672898" observedRunningTime="2026-04-21 17:33:36.032557317 +0000 UTC m=+19.754672291" watchObservedRunningTime="2026-04-21 17:33:36.032596103 +0000 UTC m=+19.754711078" Apr 21 17:33:36.078264 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:36.078164 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bvgql" podStartSLOduration=1.65683457 podStartE2EDuration="19.078148546s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.076172587 +0000 UTC m=+1.798287543" lastFinishedPulling="2026-04-21 17:33:35.497486547 +0000 UTC m=+19.219601519" observedRunningTime="2026-04-21 17:33:36.07794531 +0000 UTC m=+19.800060287" watchObservedRunningTime="2026-04-21 17:33:36.078148546 +0000 UTC m=+19.800263521" Apr 21 17:33:37.009468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.009218 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" event={"ID":"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3","Type":"ContainerStarted","Data":"176265a039975d11c8032869dc9c74e84b7548006116df17227e8f6f23a86348"} Apr 21 17:33:37.010903 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.010870 2578 generic.go:358] "Generic (PLEG): container finished" podID="c70a5efa-9e1c-4b04-b2df-fa59050fcd7e" containerID="fdc6287e8f6c73c5a5f84884985d1cf0e72425c5d87f33d2b4b0b3870f11bf2e" exitCode=0 Apr 21 17:33:37.011045 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.010970 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerDied","Data":"fdc6287e8f6c73c5a5f84884985d1cf0e72425c5d87f33d2b4b0b3870f11bf2e"} Apr 21 17:33:37.012761 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.012735 2578 generic.go:358] "Generic (PLEG): container finished" podID="447b22b2a1623dee81a3ca7a095362cd" containerID="8d0b25d7744804ce558a188cd9a37d01422e199490d42e62a689a3f47faded22" exitCode=0 Apr 21 17:33:37.012854 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.012800 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" event={"ID":"447b22b2a1623dee81a3ca7a095362cd","Type":"ContainerDied","Data":"8d0b25d7744804ce558a188cd9a37d01422e199490d42e62a689a3f47faded22"} Apr 21 17:33:37.012960 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.012945 2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" Apr 21 17:33:37.016589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.016564 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vfwzr" event={"ID":"b36fb126-3aba-45e0-8d63-ad3846dcc93a","Type":"ContainerStarted","Data":"00b65a1521e1ddadb3d20771999aa892eb2ef5459048b4853e38029e0a1ea9a9"} Apr 21 17:33:37.018178 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.018150 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msdfg" event={"ID":"38857837-ebee-4764-9e74-c45e932e9f87","Type":"ContainerStarted","Data":"4c6e42440afc55bf1dfccff5b43986327f8d598292e5f0ec16bcd9823d4bd0c9"} Apr 21 17:33:37.019775 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.019613 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vcxwk" event={"ID":"547c1477-65a3-4469-8fb5-5700ad4cd216","Type":"ContainerStarted","Data":"4b2982c2eb64f57d595b9af8e2c0a7e578df065f178e0c9844a99ee4f2a32b60"} Apr 21 17:33:37.021187 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.021164 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n859" event={"ID":"951bbb2d-337a-4723-9e9c-ef08f471f1fa","Type":"ContainerStarted","Data":"3ca58469af55f6c212586a7b61173ea9fadbaaf6baca0a13c874f59febfb9d97"} Apr 21 17:33:37.023787 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.023764 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:33:37.024185 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.024097 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"8146685e13cb1ff0d4d6958a633a02283dbe5e562ab3c13fe9e4aea61d2aa02e"} Apr 21 17:33:37.061444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.061387 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-230.ec2.internal" podStartSLOduration=20.06137247 podStartE2EDuration="20.06137247s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-21 17:33:36.115325029 +0000 UTC m=+19.837440007" watchObservedRunningTime="2026-04-21 17:33:37.06137247 +0000 UTC m=+20.783487444" Apr 21 17:33:37.061622 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.061607 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 21 17:33:37.061956 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.061935 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal"] Apr 21 17:33:37.165347 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.165303 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vcxwk" podStartSLOduration=2.762682785 podStartE2EDuration="20.165288176s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.094923923 +0000 UTC m=+1.817038875" lastFinishedPulling="2026-04-21 17:33:35.497529315 +0000 UTC m=+19.219644266" observedRunningTime="2026-04-21 17:33:37.11974884 +0000 UTC m=+20.841863816" watchObservedRunningTime="2026-04-21 17:33:37.165288176 +0000 UTC m=+20.887403191" Apr 21 17:33:37.165530 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.165495 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-msdfg" podStartSLOduration=2.94194455 podStartE2EDuration="20.165484821s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.099354941 +0000 UTC m=+1.821469896" lastFinishedPulling="2026-04-21 17:33:35.322895201 +0000 UTC m=+19.045010167" observedRunningTime="2026-04-21 17:33:37.164698242 +0000 UTC m=+20.886813227" watchObservedRunningTime="2026-04-21 17:33:37.165484821 +0000 UTC m=+20.887599795" Apr 21 17:33:37.202572 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.202535 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vfwzr" podStartSLOduration=7.312697519 podStartE2EDuration="20.202518306s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.106510506 +0000 UTC m=+1.828625458" lastFinishedPulling="2026-04-21 17:33:30.996331277 +0000 UTC m=+14.718446245" observedRunningTime="2026-04-21 17:33:37.202166242 +0000 UTC m=+20.924281215" watchObservedRunningTime="2026-04-21 17:33:37.202518306 +0000 UTC m=+20.924633277" Apr 21 17:33:37.225002 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.224983 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Apr 21 17:33:37.239909 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.239863 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-5n859" podStartSLOduration=3.239847295 podStartE2EDuration="3.239847295s" podCreationTimestamp="2026-04-21 17:33:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-21 17:33:37.239513873 +0000 UTC m=+20.961628846" watchObservedRunningTime="2026-04-21 17:33:37.239847295 +0000 UTC m=+20.961962269" Apr 21 17:33:37.762959 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.762912 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:37.763149 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:37.763060 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:37.763149 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:37.763122 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret podName:dcf8f14a-09d1-4071-94df-ba4064d021a4 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:41.763103371 +0000 UTC m=+25.485218324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret") pod "global-pull-secret-syncer-8dcr9" (UID: "dcf8f14a-09d1-4071-94df-ba4064d021a4") : object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:37.777136 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.777026 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-21T17:33:37.224996546Z","UUID":"5c73160b-1a26-427f-b386-bb655c10f841","Handler":null,"Name":"","Endpoint":""} Apr 21 17:33:37.780896 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.780872 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Apr 21 17:33:37.780896 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.780901 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Apr 21 17:33:37.797915 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.797774 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:37.797915 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.797775 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:37.797915 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:37.797913 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:37.798155 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:37.797883 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:37.798155 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:37.798011 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:37.798155 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:37.798116 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:38.027916 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:38.027832 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" event={"ID":"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3","Type":"ContainerStarted","Data":"8644b1d7d4096a9e9a65d01216526d9cc108bd081f959d92553a4464653f2936"} Apr 21 17:33:38.030023 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:38.029981 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" event={"ID":"447b22b2a1623dee81a3ca7a095362cd","Type":"ContainerStarted","Data":"5c02cbbdd46fc5521d70f0ceea42676d35516a18ca8c8718b9aebbbcddb142f8"} Apr 21 17:33:38.052998 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:38.052945 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-230.ec2.internal" podStartSLOduration=1.052928378 podStartE2EDuration="1.052928378s" podCreationTimestamp="2026-04-21 17:33:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-21 17:33:38.052630733 +0000 UTC m=+21.774745707" watchObservedRunningTime="2026-04-21 17:33:38.052928378 +0000 UTC m=+21.775043354" Apr 21 17:33:39.034590 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:39.034565 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:33:39.035152 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:39.034940 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"7c2036cc1081337069c8932a231cb58a6b4031fe2e84f49a23c3f1e82a741f34"} Apr 21 17:33:39.037011 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:39.036986 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" event={"ID":"6555c8ac-e4fb-4eb6-a7f7-c39b8a7c2fd3","Type":"ContainerStarted","Data":"e996614ca4fc4abbff52d11a7d9743e1e913eb3cd150bdbcb1652fbdfdbd25d2"} Apr 21 17:33:39.797736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:39.797708 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:39.797925 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:39.797708 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:39.797925 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:39.797822 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:39.797925 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:39.797708 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:39.797925 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:39.797902 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:39.798088 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:39.797998 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:41.744373 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:41.744340 2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:41.744941 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:41.744864 2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:41.768868 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:41.768830 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvdpr" podStartSLOduration=4.772247427 podStartE2EDuration="24.768819166s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.06981105 +0000 UTC m=+1.791926001" lastFinishedPulling="2026-04-21 17:33:38.066382763 +0000 UTC m=+21.788497740" observedRunningTime="2026-04-21 17:33:39.092355593 +0000 UTC m=+22.814470577" watchObservedRunningTime="2026-04-21 17:33:41.768819166 +0000 UTC m=+25.490934140" Apr 21 17:33:41.793372 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:41.793346 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:41.793503 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:41.793489 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:41.793551 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:41.793542 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret podName:dcf8f14a-09d1-4071-94df-ba4064d021a4 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:49.793526901 +0000 UTC m=+33.515641864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret") pod "global-pull-secret-syncer-8dcr9" (UID: "dcf8f14a-09d1-4071-94df-ba4064d021a4") : object "kube-system"/"original-pull-secret" not registered Apr 21 17:33:41.797442 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:41.797428 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:41.797507 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:41.797443 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:41.797507 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:41.797455 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:41.797572 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:41.797520 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:41.797603 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:41.797576 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:41.797660 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:41.797641 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:42.044570 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.044497 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:33:42.044878 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.044851 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"6513d8afe73161590d17744625db124ddb3ed66b72ec9e4cb7c3a0b1cb8d0827"} Apr 21 17:33:42.045117 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.045083 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:42.045117 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.045113 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:42.045225 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.045125 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:42.045369 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.045348 2578 scope.go:117] "RemoveContainer" containerID="35fe362212d85b9c3a0be6644d8e7e4714c35cea6ccf1a37aec5944320df73d1" Apr 21 17:33:42.046701 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.046675 2578 generic.go:358] "Generic (PLEG): container finished" podID="c70a5efa-9e1c-4b04-b2df-fa59050fcd7e" containerID="22efc936063514c12ca89113b30f6c64e60433b199c65e0cbd12b063e785222e" exitCode=0 Apr 21 17:33:42.046782 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.046755 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerDied","Data":"22efc936063514c12ca89113b30f6c64e60433b199c65e0cbd12b063e785222e"} Apr 21 17:33:42.047039 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.047000 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:42.047668 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.047637 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vfwzr" Apr 21 17:33:42.060652 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.060489 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:42.061707 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:42.061692 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:33:43.052615 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.052542 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:33:43.052983 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.052913 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" event={"ID":"abdbee8d-8443-4ccd-a3d5-ef918b3fc39a","Type":"ContainerStarted","Data":"e2aeaf894dd662357fee6e4c89f9582250cedb5ec944c5e0560d2868abe2713c"} Apr 21 17:33:43.054915 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.054894 2578 generic.go:358] "Generic (PLEG): container finished" podID="c70a5efa-9e1c-4b04-b2df-fa59050fcd7e" containerID="dc193e4f1eee08ff2aa73d830e99b9f866dfb455b5810e1b6ca32f8adbeba9c9" exitCode=0 Apr 21 17:33:43.055025 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.054978 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerDied","Data":"dc193e4f1eee08ff2aa73d830e99b9f866dfb455b5810e1b6ca32f8adbeba9c9"} Apr 21 17:33:43.095051 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.095008 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" podStartSLOduration=8.647622713 podStartE2EDuration="26.094995695s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.11080623 +0000 UTC m=+1.832921183" lastFinishedPulling="2026-04-21 17:33:35.558179208 +0000 UTC m=+19.280294165" observedRunningTime="2026-04-21 17:33:43.091939286 +0000 UTC m=+26.814054259" watchObservedRunningTime="2026-04-21 17:33:43.094995695 +0000 UTC m=+26.817110669" Apr 21 17:33:43.286626 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.286593 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dcr9"] Apr 21 17:33:43.286760 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.286713 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:43.286819 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:43.286799 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:43.300030 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.300002 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5d95f"] Apr 21 17:33:43.300136 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.300101 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:43.300190 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:43.300173 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:43.303151 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.303094 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z46hp"] Apr 21 17:33:43.303240 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:43.303184 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:43.303307 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:43.303286 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:44.058865 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:44.058776 2578 generic.go:358] "Generic (PLEG): container finished" podID="c70a5efa-9e1c-4b04-b2df-fa59050fcd7e" containerID="f2e5d24af16d83c28cc9610cecf9e2d7a880096e5ab013309f3bc389eadb3d7e" exitCode=0 Apr 21 17:33:44.059231 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:44.058871 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerDied","Data":"f2e5d24af16d83c28cc9610cecf9e2d7a880096e5ab013309f3bc389eadb3d7e"} Apr 21 17:33:44.797166 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:44.797100 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:44.797316 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:44.797220 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:44.797316 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:44.797230 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:44.797476 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:44.797340 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:44.797476 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:44.797391 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:44.797577 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:44.797471 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:46.798967 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:46.798928 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:46.799391 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:46.799015 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:46.799391 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:46.799074 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:33:46.799391 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:46.799154 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:46.799391 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:46.799256 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d95f" podUID="9959f536-1d43-45dc-a1b9-a84e76dbafa0" Apr 21 17:33:46.799391 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:46.799293 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8dcr9" podUID="dcf8f14a-09d1-4071-94df-ba4064d021a4" Apr 21 17:33:48.614204 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.614174 2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-230.ec2.internal" event="NodeReady" Apr 21 17:33:48.614631 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.614316 2578 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Apr 21 17:33:48.653957 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.653927 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f84d86b-5wqs4"] Apr 21 17:33:48.690995 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.690962 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7"] Apr 21 17:33:48.691172 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.691154 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.694604 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.694521 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Apr 21 17:33:48.694768 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.694671 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6n9k\"" Apr 21 17:33:48.694847 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.694785 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Apr 21 17:33:48.694847 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.694527 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Apr 21 17:33:48.700342 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.700320 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Apr 21 17:33:48.706489 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.706467 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4"] Apr 21 17:33:48.706607 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.706529 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:48.709316 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.709296 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hv5zp\"" Apr 21 17:33:48.709629 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.709550 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Apr 21 17:33:48.709629 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.709616 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Apr 21 17:33:48.752276 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.752245 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t"] Apr 21 17:33:48.752550 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.752526 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:48.755564 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.755432 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Apr 21 17:33:48.755564 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.755467 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Apr 21 17:33:48.755564 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.755437 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Apr 21 17:33:48.765988 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.765968 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Apr 21 17:33:48.766917 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.766900 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj"] Apr 21 17:33:48.767046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.767027 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.769689 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.769665 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Apr 21 17:33:48.785613 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.785573 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4qz45"] Apr 21 17:33:48.785749 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.785729 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:48.788571 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.788530 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Apr 21 17:33:48.788571 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.788532 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Apr 21 17:33:48.789234 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.789213 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Apr 21 17:33:48.789325 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.789265 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Apr 21 17:33:48.808921 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.808896 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7"] Apr 21 17:33:48.809043 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.808985 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:48.809105 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.808985 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:48.809105 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.809080 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:48.809473 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.809454 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:48.812798 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.812781 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f84d86b-5wqs4"] Apr 21 17:33:48.812889 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.812803 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4"] Apr 21 17:33:48.812889 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.812814 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t"] Apr 21 17:33:48.812889 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.812824 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj"] Apr 21 17:33:48.812889 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.812833 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8bv2c"] Apr 21 17:33:48.815247 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815229 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 21 17:33:48.815346 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815331 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Apr 21 17:33:48.815406 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815351 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 21 17:33:48.815406 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815389 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5lbmh\"" Apr 21 17:33:48.815523 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815451 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Apr 21 17:33:48.815523 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815389 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mlvk8\"" Apr 21 17:33:48.815523 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815455 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 21 17:33:48.815808 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815786 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pqc26\"" Apr 21 17:33:48.815890 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.815800 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 21 17:33:48.831117 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.831096 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4qz45"] Apr 21 17:33:48.831117 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.831121 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bv2c"] Apr 21 17:33:48.831283 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.831231 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:48.833888 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.833871 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4fbh6\"" Apr 21 17:33:48.833974 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.833871 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Apr 21 17:33:48.833974 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.833919 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Apr 21 17:33:48.834221 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.834202 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Apr 21 17:33:48.850575 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dece6707-af2b-4518-b0f4-95ec5f993643-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:48.850671 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a543a5c-3a96-425b-8771-bc2371809c86-tmp\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.850671 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-bound-sa-token\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.850671 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-image-registry-private-configuration\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.850671 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-installation-pull-secrets\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.850850 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:48.850850 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-registry-certificates\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.850850 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmm6\" (UniqueName: \"kubernetes.io/projected/7a543a5c-3a96-425b-8771-bc2371809c86-kube-api-access-5vmm6\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.850850 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7a543a5c-3a96-425b-8771-bc2371809c86-klusterlet-config\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.851022 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b62c08-2675-4759-887b-5dd01667868c-ca-trust-extracted\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.851022 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-trusted-ca\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.851022 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.850937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75263093-7984-4bc3-b1d9-b80346987ea2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-546698bbdf-pl4m4\" (UID: \"75263093-7984-4bc3-b1d9-b80346987ea2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:48.851022 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.851007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22hr\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-kube-api-access-r22hr\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.851183 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.851039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljw5\" (UniqueName: \"kubernetes.io/projected/75263093-7984-4bc3-b1d9-b80346987ea2-kube-api-access-kljw5\") pod \"managed-serviceaccount-addon-agent-546698bbdf-pl4m4\" (UID: \"75263093-7984-4bc3-b1d9-b80346987ea2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:48.851183 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.851103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952355 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952271 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952355 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:48.952594 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952378 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-image-registry-private-configuration\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952594 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952409 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-installation-pull-secrets\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952594 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:48.952448 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:33:48.952594 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:48.952471 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:33:48.952594 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmkd\" (UniqueName: \"kubernetes.io/projected/1b8ca466-7111-427f-891f-efc7ebe9d92d-kube-api-access-5jmkd\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:48.952594 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:48.952549 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:33:49.45252461 +0000 UTC m=+33.174639569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:33:48.952594 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/413bc0c4-35c9-4de3-8aff-47a10cfd1322-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj79v\" (UniqueName: \"kubernetes.io/projected/413bc0c4-35c9-4de3-8aff-47a10cfd1322-kube-api-access-nj79v\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbt6\" (UniqueName: \"kubernetes.io/projected/137fa6ca-d790-417d-bc67-5a19c38af051-kube-api-access-rfbt6\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952694 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b62c08-2675-4759-887b-5dd01667868c-ca-trust-extracted\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952718 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-trusted-ca\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952741 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75263093-7984-4bc3-b1d9-b80346987ea2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-546698bbdf-pl4m4\" (UID: \"75263093-7984-4bc3-b1d9-b80346987ea2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952801 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r22hr\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-kube-api-access-r22hr\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952845 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dece6707-af2b-4518-b0f4-95ec5f993643-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952910 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-bound-sa-token\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.952954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952939 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a543a5c-3a96-425b-8771-bc2371809c86-tmp\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.953461 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.952970 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:48.953461 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953013 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-registry-certificates\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.953461 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/137fa6ca-d790-417d-bc67-5a19c38af051-config-volume\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:48.953461 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953100 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmm6\" (UniqueName: \"kubernetes.io/projected/7a543a5c-3a96-425b-8771-bc2371809c86-kube-api-access-5vmm6\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.953461 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b62c08-2675-4759-887b-5dd01667868c-ca-trust-extracted\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.953706 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:48.953484 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:33:48.953706 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a543a5c-3a96-425b-8771-bc2371809c86-tmp\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.953706 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:48.953546 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:49.453526093 +0000 UTC m=+33.175641049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:33:48.953881 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-ca\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:48.953934 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-hub\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:48.953934 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:48.954026 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953954 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7a543a5c-3a96-425b-8771-bc2371809c86-klusterlet-config\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.954026 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.953982 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kljw5\" (UniqueName: \"kubernetes.io/projected/75263093-7984-4bc3-b1d9-b80346987ea2-kube-api-access-kljw5\") pod \"managed-serviceaccount-addon-agent-546698bbdf-pl4m4\" (UID: \"75263093-7984-4bc3-b1d9-b80346987ea2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:48.954026 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.954010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/137fa6ca-d790-417d-bc67-5a19c38af051-tmp-dir\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:48.954218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.954085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-trusted-ca\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.954218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.954116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dece6707-af2b-4518-b0f4-95ec5f993643-nginx-conf\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:48.954218 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.954149 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-registry-certificates\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.957111 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.957091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-installation-pull-secrets\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.957206 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.957109 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-image-registry-private-configuration\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.957277 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.957256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7a543a5c-3a96-425b-8771-bc2371809c86-klusterlet-config\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:48.957443 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.957405 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75263093-7984-4bc3-b1d9-b80346987ea2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-546698bbdf-pl4m4\" (UID: \"75263093-7984-4bc3-b1d9-b80346987ea2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:48.964877 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.964853 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljw5\" (UniqueName: \"kubernetes.io/projected/75263093-7984-4bc3-b1d9-b80346987ea2-kube-api-access-kljw5\") pod \"managed-serviceaccount-addon-agent-546698bbdf-pl4m4\" (UID: \"75263093-7984-4bc3-b1d9-b80346987ea2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:48.964969 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.964878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-bound-sa-token\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.965035 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.964973 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22hr\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-kube-api-access-r22hr\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:48.966527 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:48.966502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmm6\" (UniqueName: \"kubernetes.io/projected/7a543a5c-3a96-425b-8771-bc2371809c86-kube-api-access-5vmm6\") pod \"klusterlet-addon-workmgr-6788d84d6d-q6d2t\" (UID: \"7a543a5c-3a96-425b-8771-bc2371809c86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:49.055324 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055285 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-ca\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.055516 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055334 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-hub\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.055516 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055355 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.055516 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055388 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/137fa6ca-d790-417d-bc67-5a19c38af051-tmp-dir\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.055516 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055463 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.055516 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055493 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmkd\" (UniqueName: \"kubernetes.io/projected/1b8ca466-7111-427f-891f-efc7ebe9d92d-kube-api-access-5jmkd\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:49.055768 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055523 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.055768 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055549 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/413bc0c4-35c9-4de3-8aff-47a10cfd1322-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.055768 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055573 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nj79v\" (UniqueName: \"kubernetes.io/projected/413bc0c4-35c9-4de3-8aff-47a10cfd1322-kube-api-access-nj79v\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.055768 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055595 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbt6\" (UniqueName: \"kubernetes.io/projected/137fa6ca-d790-417d-bc67-5a19c38af051-kube-api-access-rfbt6\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.055768 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055628 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:49.055768 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055732 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/137fa6ca-d790-417d-bc67-5a19c38af051-config-volume\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.056048 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.055838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/137fa6ca-d790-417d-bc67-5a19c38af051-tmp-dir\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.056621 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.056221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/137fa6ca-d790-417d-bc67-5a19c38af051-config-volume\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.056621 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.056316 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:33:49.056621 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.056367 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:33:49.556350061 +0000 UTC m=+33.278465020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:33:49.056621 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.056386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/413bc0c4-35c9-4de3-8aff-47a10cfd1322-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.056621 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.056518 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:33:49.056621 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.056578 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:49.556561158 +0000 UTC m=+33.278676116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:33:49.058548 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.058528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.058660 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.058550 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-hub\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.058862 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.058841 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-ca\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.058912 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.058875 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/413bc0c4-35c9-4de3-8aff-47a10cfd1322-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.065723 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.065676 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmkd\" (UniqueName: \"kubernetes.io/projected/1b8ca466-7111-427f-891f-efc7ebe9d92d-kube-api-access-5jmkd\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:49.065831 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.065778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbt6\" (UniqueName: \"kubernetes.io/projected/137fa6ca-d790-417d-bc67-5a19c38af051-kube-api-access-rfbt6\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.066006 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.065962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj79v\" (UniqueName: \"kubernetes.io/projected/413bc0c4-35c9-4de3-8aff-47a10cfd1322-kube-api-access-nj79v\") pod \"cluster-proxy-proxy-agent-bdc6878b7-x9rkj\" (UID: \"413bc0c4-35c9-4de3-8aff-47a10cfd1322\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.072347 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.072324 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" Apr 21 17:33:49.080046 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.080026 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:33:49.098855 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.098826 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:33:49.459482 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.459437 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:49.459683 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.459495 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:49.459683 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.459598 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:33:49.459683 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.459676 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:50.459652635 +0000 UTC m=+34.181767589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:33:49.459847 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.459602 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:33:49.459847 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.459712 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:33:49.459847 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.459767 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:33:50.459750774 +0000 UTC m=+34.181865729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:33:49.560806 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.560766 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:49.560970 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.560821 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:33:49.560970 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.560882 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:49.560970 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.560936 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:33:49.560970 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.560963 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:33:49.560970 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.560965 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 21 17:33:49.561173 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.561005 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:50.56099225 +0000 UTC m=+34.283107201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:33:49.561173 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.561016 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:33:50.561010967 +0000 UTC m=+34.283125918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:33:49.561173 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:49.561026 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:34:21.561021191 +0000 UTC m=+65.283136142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : secret "metrics-daemon-secret" not found Apr 21 17:33:49.661727 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.661692 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:49.664229 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.664212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54hvt\" (UniqueName: \"kubernetes.io/projected/9959f536-1d43-45dc-a1b9-a84e76dbafa0-kube-api-access-54hvt\") pod \"network-check-target-5d95f\" (UID: \"9959f536-1d43-45dc-a1b9-a84e76dbafa0\") " pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:49.748000 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.747774 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:33:49.778804 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.778716 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t"] Apr 21 17:33:49.780106 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.780083 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4"] Apr 21 17:33:49.781193 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.781076 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj"] Apr 21 17:33:49.863585 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.863550 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:49.865944 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:49.865918 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcf8f14a-09d1-4071-94df-ba4064d021a4-original-pull-secret\") pod \"global-pull-secret-syncer-8dcr9\" (UID: \"dcf8f14a-09d1-4071-94df-ba4064d021a4\") " pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:49.875995 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:49.875972 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75263093_7984_4bc3_b1d9_b80346987ea2.slice/crio-f97c7908f0cfe0c5384ec87c508a04d92e3feefcae64dd07b0db3309c4d26d20 WatchSource:0}: Error finding container f97c7908f0cfe0c5384ec87c508a04d92e3feefcae64dd07b0db3309c4d26d20: Status 404 returned error can't find the container with id f97c7908f0cfe0c5384ec87c508a04d92e3feefcae64dd07b0db3309c4d26d20 Apr 21 17:33:49.876711 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:49.876685 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod413bc0c4_35c9_4de3_8aff_47a10cfd1322.slice/crio-37c02ee3b06f8883a0aed4d67237cf722a599b706de30ee80411b79373576c4a WatchSource:0}: Error finding container 37c02ee3b06f8883a0aed4d67237cf722a599b706de30ee80411b79373576c4a: Status 404 returned error can't find the container with id 37c02ee3b06f8883a0aed4d67237cf722a599b706de30ee80411b79373576c4a Apr 21 17:33:49.877537 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:49.877514 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a543a5c_3a96_425b_8771_bc2371809c86.slice/crio-8bcfae7f30c5032be170e4770d1c2fc411f17aaba1686f67f375ec635336dcfd WatchSource:0}: Error finding container 8bcfae7f30c5032be170e4770d1c2fc411f17aaba1686f67f375ec635336dcfd: Status 404 returned error can't find the container with id 8bcfae7f30c5032be170e4770d1c2fc411f17aaba1686f67f375ec635336dcfd Apr 21 17:33:50.029090 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.028814 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5d95f"] Apr 21 17:33:50.031963 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:50.031939 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9959f536_1d43_45dc_a1b9_a84e76dbafa0.slice/crio-93eff6aad6519e9ed124c02302d6258b6398c2c4f2598c3422d7cdfc9a44dfe4 WatchSource:0}: Error finding container 93eff6aad6519e9ed124c02302d6258b6398c2c4f2598c3422d7cdfc9a44dfe4: Status 404 returned error can't find the container with id 93eff6aad6519e9ed124c02302d6258b6398c2c4f2598c3422d7cdfc9a44dfe4 Apr 21 17:33:50.033137 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.033113 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8dcr9" Apr 21 17:33:50.073091 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.073061 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" event={"ID":"7a543a5c-3a96-425b-8771-bc2371809c86","Type":"ContainerStarted","Data":"8bcfae7f30c5032be170e4770d1c2fc411f17aaba1686f67f375ec635336dcfd"} Apr 21 17:33:50.073965 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.073933 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" event={"ID":"75263093-7984-4bc3-b1d9-b80346987ea2","Type":"ContainerStarted","Data":"f97c7908f0cfe0c5384ec87c508a04d92e3feefcae64dd07b0db3309c4d26d20"} Apr 21 17:33:50.074824 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.074810 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" event={"ID":"413bc0c4-35c9-4de3-8aff-47a10cfd1322","Type":"ContainerStarted","Data":"37c02ee3b06f8883a0aed4d67237cf722a599b706de30ee80411b79373576c4a"} Apr 21 17:33:50.075668 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.075652 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5d95f" event={"ID":"9959f536-1d43-45dc-a1b9-a84e76dbafa0","Type":"ContainerStarted","Data":"93eff6aad6519e9ed124c02302d6258b6398c2c4f2598c3422d7cdfc9a44dfe4"} Apr 21 17:33:50.157869 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.157722 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8dcr9"] Apr 21 17:33:50.160698 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:33:50.160673 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf8f14a_09d1_4071_94df_ba4064d021a4.slice/crio-167c3615a0b28bacec23149080a492361b170e89d80e40646aa5b03c3807312e WatchSource:0}: Error finding container 167c3615a0b28bacec23149080a492361b170e89d80e40646aa5b03c3807312e: Status 404 returned error can't find the container with id 167c3615a0b28bacec23149080a492361b170e89d80e40646aa5b03c3807312e Apr 21 17:33:50.468534 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.468493 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:50.468741 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.468600 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:50.468741 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.468643 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:33:50.468741 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.468664 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:33:50.468741 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.468678 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:33:50.468741 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.468722 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:33:52.468705656 +0000 UTC m=+36.190820607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:33:50.468741 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.468735 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:52.468729001 +0000 UTC m=+36.190843953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:33:50.569188 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.569072 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:50.569188 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:50.569128 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:50.569429 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.569293 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:33:50.569429 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.569352 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:33:52.569333312 +0000 UTC m=+36.291448271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:33:50.569584 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.569561 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:33:50.569697 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:50.569670 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:52.569616095 +0000 UTC m=+36.291731071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:33:51.089109 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:51.089037 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dcr9" event={"ID":"dcf8f14a-09d1-4071-94df-ba4064d021a4","Type":"ContainerStarted","Data":"167c3615a0b28bacec23149080a492361b170e89d80e40646aa5b03c3807312e"} Apr 21 17:33:51.097462 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:51.097428 2578 generic.go:358] "Generic (PLEG): container finished" podID="c70a5efa-9e1c-4b04-b2df-fa59050fcd7e" containerID="6cd6454801e635c271b84976cbf66cd87950f812d4599ba30b89afd43933c354" exitCode=0 Apr 21 17:33:51.097610 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:51.097476 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerDied","Data":"6cd6454801e635c271b84976cbf66cd87950f812d4599ba30b89afd43933c354"} Apr 21 17:33:52.120574 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:52.119678 2578 generic.go:358] "Generic (PLEG): container finished" podID="c70a5efa-9e1c-4b04-b2df-fa59050fcd7e" containerID="e954511736b544a2acaa33a556a90a6f173db24eba41b24aaf717c87c8312e25" exitCode=0 Apr 21 17:33:52.120574 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:52.119734 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerDied","Data":"e954511736b544a2acaa33a556a90a6f173db24eba41b24aaf717c87c8312e25"} Apr 21 17:33:52.488615 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:52.487871 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:52.488615 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:52.487928 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:52.488615 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.488050 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:33:52.488615 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.488064 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:33:52.488615 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.488124 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:33:56.488104249 +0000 UTC m=+40.210219204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:33:52.488615 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.488201 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:33:52.488615 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.488235 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:56.48822404 +0000 UTC m=+40.210338992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:33:52.589993 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:52.588931 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:52.589993 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:52.589224 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:52.589993 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.589377 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:33:52.589993 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.589454 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:33:56.589433288 +0000 UTC m=+40.311548245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:33:52.589993 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.589857 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:33:52.589993 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:52.589924 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:33:56.589894992 +0000 UTC m=+40.312009945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:33:53.128767 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:53.128359 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" event={"ID":"c70a5efa-9e1c-4b04-b2df-fa59050fcd7e","Type":"ContainerStarted","Data":"7f0a3129ebfba8d5e1fdf1620e5d4da0385205023001e77a5829f09f0a0e0938"} Apr 21 17:33:56.525694 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:56.525609 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:33:56.526290 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:56.525722 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:33:56.526290 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.525773 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:33:56.526290 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.525858 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:34:04.525833963 +0000 UTC m=+48.247948915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:33:56.526290 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.525883 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:33:56.526290 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.525902 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:33:56.526290 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.525962 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:34:04.525945542 +0000 UTC m=+48.248060502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:33:56.627080 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:56.627040 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:33:56.627255 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:56.627092 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:33:56.627255 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.627213 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:33:56.627255 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.627231 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:33:56.627395 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.627295 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:34:04.62727137 +0000 UTC m=+48.349386337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:33:56.627395 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:33:56.627315 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:34:04.627305301 +0000 UTC m=+48.349420259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:33:56.828137 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:33:56.828045 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9fvp6" podStartSLOduration=8.944810567 podStartE2EDuration="40.828030335s" podCreationTimestamp="2026-04-21 17:33:16 +0000 UTC" firstStartedPulling="2026-04-21 17:33:18.046072996 +0000 UTC m=+1.768187962" lastFinishedPulling="2026-04-21 17:33:49.929292759 +0000 UTC m=+33.651407730" observedRunningTime="2026-04-21 17:33:53.157507548 +0000 UTC m=+36.879622514" watchObservedRunningTime="2026-04-21 17:33:56.828030335 +0000 UTC m=+40.550145308" Apr 21 17:34:00.146111 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.146070 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8dcr9" event={"ID":"dcf8f14a-09d1-4071-94df-ba4064d021a4","Type":"ContainerStarted","Data":"0b35adb757d710cf10a717cf6933a6b4d19f1acb236c1ee69afb275545926542"} Apr 21 17:34:00.147467 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.147442 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5d95f" event={"ID":"9959f536-1d43-45dc-a1b9-a84e76dbafa0","Type":"ContainerStarted","Data":"acf034b43aaa41729fd2d7ca1a39cf929456dd9d26382957eba6e6a165a07445"} Apr 21 17:34:00.147578 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.147562 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:34:00.148750 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.148730 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" event={"ID":"7a543a5c-3a96-425b-8771-bc2371809c86","Type":"ContainerStarted","Data":"71f274b29138be747c8c3fceee95537d79e10e80e9fdfef612a33c083db1c4db"} Apr 21 17:34:00.148929 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.148913 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:34:00.150100 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.150072 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" event={"ID":"75263093-7984-4bc3-b1d9-b80346987ea2","Type":"ContainerStarted","Data":"22dc87eee689b36fca6c865f4a31ecc77586d904100af1d369fe2ec5756f622c"} Apr 21 17:34:00.150694 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.150678 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:34:00.151395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.151378 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" event={"ID":"413bc0c4-35c9-4de3-8aff-47a10cfd1322","Type":"ContainerStarted","Data":"20dd5ec96c07b3a6ccae7335ca9859adcb1384b0b18768e8afc8e90795c1df4b"} Apr 21 17:34:00.166629 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.166534 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8dcr9" podStartSLOduration=17.705048525 podStartE2EDuration="27.166519231s" podCreationTimestamp="2026-04-21 17:33:33 +0000 UTC" firstStartedPulling="2026-04-21 17:33:50.163108619 +0000 UTC m=+33.885223571" lastFinishedPulling="2026-04-21 17:33:59.624579325 +0000 UTC m=+43.346694277" observedRunningTime="2026-04-21 17:34:00.16603434 +0000 UTC m=+43.888149335" watchObservedRunningTime="2026-04-21 17:34:00.166519231 +0000 UTC m=+43.888634216" Apr 21 17:34:00.187654 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.187609 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" podStartSLOduration=30.477627709 podStartE2EDuration="40.187595314s" podCreationTimestamp="2026-04-21 17:33:20 +0000 UTC" firstStartedPulling="2026-04-21 17:33:49.903186341 +0000 UTC m=+33.625301304" lastFinishedPulling="2026-04-21 17:33:59.613153943 +0000 UTC m=+43.335268909" observedRunningTime="2026-04-21 17:34:00.186588418 +0000 UTC m=+43.908703391" watchObservedRunningTime="2026-04-21 17:34:00.187595314 +0000 UTC m=+43.909710325" Apr 21 17:34:00.204249 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.204204 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5d95f" podStartSLOduration=33.624682483 podStartE2EDuration="43.204190825s" podCreationTimestamp="2026-04-21 17:33:17 +0000 UTC" firstStartedPulling="2026-04-21 17:33:50.034219982 +0000 UTC m=+33.756334947" lastFinishedPulling="2026-04-21 17:33:59.613728324 +0000 UTC m=+43.335843289" observedRunningTime="2026-04-21 17:34:00.20405625 +0000 UTC m=+43.926171214" watchObservedRunningTime="2026-04-21 17:34:00.204190825 +0000 UTC m=+43.926305796" Apr 21 17:34:00.227351 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:00.227297 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" podStartSLOduration=30.517367303 podStartE2EDuration="40.227282701s" podCreationTimestamp="2026-04-21 17:33:20 +0000 UTC" firstStartedPulling="2026-04-21 17:33:49.90324411 +0000 UTC m=+33.625359061" lastFinishedPulling="2026-04-21 17:33:59.613159504 +0000 UTC m=+43.335274459" observedRunningTime="2026-04-21 17:34:00.226153149 +0000 UTC m=+43.948268123" watchObservedRunningTime="2026-04-21 17:34:00.227282701 +0000 UTC m=+43.949397676" Apr 21 17:34:03.161530 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:03.161491 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" event={"ID":"413bc0c4-35c9-4de3-8aff-47a10cfd1322","Type":"ContainerStarted","Data":"58fdcfd3fbcba8eeca21e850d6ab028435cd068b5892aa44e77a7f6a51b282ff"} Apr 21 17:34:03.161530 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:03.161532 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" event={"ID":"413bc0c4-35c9-4de3-8aff-47a10cfd1322","Type":"ContainerStarted","Data":"eb3b7a3a7246a3b56d31c1de9c2a5601772a9e31f4c6b4ec1e5b7e555c20d280"} Apr 21 17:34:03.183003 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:03.182956 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" podStartSLOduration=30.688090105 podStartE2EDuration="43.182942173s" podCreationTimestamp="2026-04-21 17:33:20 +0000 UTC" firstStartedPulling="2026-04-21 17:33:49.903187265 +0000 UTC m=+33.625302217" lastFinishedPulling="2026-04-21 17:34:02.398039332 +0000 UTC m=+46.120154285" observedRunningTime="2026-04-21 17:34:03.181100036 +0000 UTC m=+46.903215010" watchObservedRunningTime="2026-04-21 17:34:03.182942173 +0000 UTC m=+46.905057188" Apr 21 17:34:04.593816 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:04.593777 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:34:04.594208 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:04.593827 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:34:04.594208 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.593921 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:34:04.594208 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.593936 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:34:04.594208 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.593920 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:34:04.594208 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.593983 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:34:20.593965529 +0000 UTC m=+64.316080481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:34:04.594208 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.594023 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:34:20.594008262 +0000 UTC m=+64.316123217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:34:04.694641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:04.694611 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:34:04.694806 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:04.694648 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:34:04.694806 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.694745 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:34:04.694806 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.694774 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:34:04.694806 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.694798 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:34:20.694783826 +0000 UTC m=+64.416898778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:34:04.694958 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:04.694814 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:34:20.694803947 +0000 UTC m=+64.416918899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:34:14.074012 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:14.073983 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms5gf" Apr 21 17:34:20.609781 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:20.609737 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:34:20.610205 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:20.609857 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:34:20.610205 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.609884 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:34:20.610205 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.609902 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:34:20.610205 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.609952 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:34:20.610205 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.609953 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:34:52.609939348 +0000 UTC m=+96.332054300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:34:20.610205 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.610004 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:34:52.609989304 +0000 UTC m=+96.332104256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:34:20.710511 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:20.710478 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:34:20.710650 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:20.710525 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:34:20.710650 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.710631 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:34:20.710722 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.710691 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:34:20.710754 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.710696 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:34:52.71067981 +0000 UTC m=+96.432794785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:34:20.710794 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:20.710758 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:34:52.710745591 +0000 UTC m=+96.432860542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:34:21.617151 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:21.617115 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:34:21.617643 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:21.617284 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 21 17:34:21.617643 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:21.617367 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:35:25.617347121 +0000 UTC m=+129.339462074 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : secret "metrics-daemon-secret" not found Apr 21 17:34:31.159267 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:31.159139 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5d95f" Apr 21 17:34:52.665669 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:52.665618 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:34:52.665669 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:52.665677 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:34:52.666103 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.665773 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:34:52.666103 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.665801 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:34:52.666103 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.665812 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:34:52.666103 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.665854 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:35:56.665838222 +0000 UTC m=+160.387953174 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:34:52.666103 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.665870 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:35:56.665863349 +0000 UTC m=+160.387978302 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:34:52.766559 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:52.766519 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:34:52.766559 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:34:52.766566 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:34:52.766726 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.766664 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:34:52.766726 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.766665 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:34:52.766726 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.766714 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:35:56.76669986 +0000 UTC m=+160.488814811 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:34:52.766726 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:34:52.766726 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:35:56.766720179 +0000 UTC m=+160.488835131 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:35:25.714914 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:25.714876 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:35:25.715400 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:25.715022 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 21 17:35:25.715400 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:25.715091 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs podName:cbb61ff1-0bdf-4220-99ee-6f860546dc41 nodeName:}" failed. No retries permitted until 2026-04-21 17:37:27.715075541 +0000 UTC m=+251.437190498 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs") pod "network-metrics-daemon-z46hp" (UID: "cbb61ff1-0bdf-4220-99ee-6f860546dc41") : secret "metrics-daemon-secret" not found Apr 21 17:35:45.385732 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:45.385704 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n859_951bbb2d-337a-4723-9e9c-ef08f471f1fa/dns-node-resolver/0.log" Apr 21 17:35:46.585164 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:46.585137 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vcxwk_547c1477-65a3-4469-8fb5-5700ad4cd216/node-ca/0.log" Apr 21 17:35:51.704205 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:51.704163 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" podUID="b1b62c08-2675-4759-887b-5dd01667868c" Apr 21 17:35:51.716451 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:51.716409 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" podUID="dece6707-af2b-4518-b0f4-95ec5f993643" Apr 21 17:35:51.840667 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:51.840626 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-z46hp" podUID="cbb61ff1-0bdf-4220-99ee-6f860546dc41" Apr 21 17:35:51.856009 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:51.855987 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4qz45" podUID="137fa6ca-d790-417d-bc67-5a19c38af051" Apr 21 17:35:51.862128 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:51.862096 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-8bv2c" podUID="1b8ca466-7111-427f-891f-efc7ebe9d92d" Apr 21 17:35:52.412064 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:52.412031 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:35:52.412262 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:52.412036 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qz45" Apr 21 17:35:52.412262 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:52.412036 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:35:56.752042 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:56.751996 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:35:56.752583 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:56.752087 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") pod \"image-registry-68f84d86b-5wqs4\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:35:56.752583 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.752270 2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 21 17:35:56.752583 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.752298 2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f84d86b-5wqs4: secret "image-registry-tls" not found Apr 21 17:35:56.752583 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.752306 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 21 17:35:56.752583 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.752505 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls podName:b1b62c08-2675-4759-887b-5dd01667868c nodeName:}" failed. No retries permitted until 2026-04-21 17:37:58.752473793 +0000 UTC m=+282.474588750 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls") pod "image-registry-68f84d86b-5wqs4" (UID: "b1b62c08-2675-4759-887b-5dd01667868c") : secret "image-registry-tls" not found Apr 21 17:35:56.752825 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.752608 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert podName:dece6707-af2b-4518-b0f4-95ec5f993643 nodeName:}" failed. No retries permitted until 2026-04-21 17:37:58.752588684 +0000 UTC m=+282.474703637 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert") pod "networking-console-plugin-cb95c66f6-kzkr7" (UID: "dece6707-af2b-4518-b0f4-95ec5f993643") : secret "networking-console-plugin-cert" not found Apr 21 17:35:56.852909 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:56.852878 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:35:56.852909 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:56.852914 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:35:56.853142 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.853003 2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 21 17:35:56.853142 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.853020 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 21 17:35:56.853142 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.853056 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert podName:1b8ca466-7111-427f-891f-efc7ebe9d92d nodeName:}" failed. No retries permitted until 2026-04-21 17:37:58.853042942 +0000 UTC m=+282.575157893 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert") pod "ingress-canary-8bv2c" (UID: "1b8ca466-7111-427f-891f-efc7ebe9d92d") : secret "canary-serving-cert" not found Apr 21 17:35:56.853142 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:56.853106 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls podName:137fa6ca-d790-417d-bc67-5a19c38af051 nodeName:}" failed. No retries permitted until 2026-04-21 17:37:58.853089683 +0000 UTC m=+282.575204653 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls") pod "dns-default-4qz45" (UID: "137fa6ca-d790-417d-bc67-5a19c38af051") : secret "dns-default-metrics-tls" not found Apr 21 17:35:57.007633 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.007568 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tz8cg"] Apr 21 17:35:57.010505 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.010485 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.012901 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.012876 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Apr 21 17:35:57.013008 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.012926 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Apr 21 17:35:57.013008 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.012944 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Apr 21 17:35:57.013188 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.013175 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Apr 21 17:35:57.014091 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.014075 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-55shw\"" Apr 21 17:35:57.020491 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.020470 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tz8cg"] Apr 21 17:35:57.155589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.155551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/033302de-ab2b-4b50-8bf5-bea5f1870198-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.155589 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.155600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.155798 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.155659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/033302de-ab2b-4b50-8bf5-bea5f1870198-crio-socket\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.155798 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.155696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/033302de-ab2b-4b50-8bf5-bea5f1870198-data-volume\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.155798 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.155754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjwz\" (UniqueName: \"kubernetes.io/projected/033302de-ab2b-4b50-8bf5-bea5f1870198-kube-api-access-zfjwz\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256033 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.255993 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjwz\" (UniqueName: \"kubernetes.io/projected/033302de-ab2b-4b50-8bf5-bea5f1870198-kube-api-access-zfjwz\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256165 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.256048 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/033302de-ab2b-4b50-8bf5-bea5f1870198-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256165 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.256080 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256165 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.256109 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/033302de-ab2b-4b50-8bf5-bea5f1870198-crio-socket\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256165 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.256133 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/033302de-ab2b-4b50-8bf5-bea5f1870198-data-volume\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256314 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:57.256226 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Apr 21 17:35:57.256314 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:57.256293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls podName:033302de-ab2b-4b50-8bf5-bea5f1870198 nodeName:}" failed. No retries permitted until 2026-04-21 17:35:57.756274767 +0000 UTC m=+161.478389719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz8cg" (UID: "033302de-ab2b-4b50-8bf5-bea5f1870198") : secret "insights-runtime-extractor-tls" not found Apr 21 17:35:57.256400 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.256357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/033302de-ab2b-4b50-8bf5-bea5f1870198-crio-socket\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256536 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.256520 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/033302de-ab2b-4b50-8bf5-bea5f1870198-data-volume\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.256623 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.256607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/033302de-ab2b-4b50-8bf5-bea5f1870198-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.266761 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.266704 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjwz\" (UniqueName: \"kubernetes.io/projected/033302de-ab2b-4b50-8bf5-bea5f1870198-kube-api-access-zfjwz\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.759682 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:57.759635 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:57.760058 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:57.759784 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Apr 21 17:35:57.760058 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:57.759844 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls podName:033302de-ab2b-4b50-8bf5-bea5f1870198 nodeName:}" failed. No retries permitted until 2026-04-21 17:35:58.759827167 +0000 UTC m=+162.481942120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz8cg" (UID: "033302de-ab2b-4b50-8bf5-bea5f1870198") : secret "insights-runtime-extractor-tls" not found Apr 21 17:35:58.768713 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:35:58.768679 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:35:58.769079 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:58.768822 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Apr 21 17:35:58.769079 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:35:58.768888 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls podName:033302de-ab2b-4b50-8bf5-bea5f1870198 nodeName:}" failed. No retries permitted until 2026-04-21 17:36:00.768872815 +0000 UTC m=+164.490987767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz8cg" (UID: "033302de-ab2b-4b50-8bf5-bea5f1870198") : secret "insights-runtime-extractor-tls" not found Apr 21 17:36:00.149982 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.149874 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" podUID="7a543a5c-3a96-425b-8771-bc2371809c86" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused" Apr 21 17:36:00.431300 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.431209 2578 generic.go:358] "Generic (PLEG): container finished" podID="7a543a5c-3a96-425b-8771-bc2371809c86" containerID="71f274b29138be747c8c3fceee95537d79e10e80e9fdfef612a33c083db1c4db" exitCode=1 Apr 21 17:36:00.431300 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.431276 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" event={"ID":"7a543a5c-3a96-425b-8771-bc2371809c86","Type":"ContainerDied","Data":"71f274b29138be747c8c3fceee95537d79e10e80e9fdfef612a33c083db1c4db"} Apr 21 17:36:00.431645 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.431625 2578 scope.go:117] "RemoveContainer" containerID="71f274b29138be747c8c3fceee95537d79e10e80e9fdfef612a33c083db1c4db" Apr 21 17:36:00.432627 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.432605 2578 generic.go:358] "Generic (PLEG): container finished" podID="75263093-7984-4bc3-b1d9-b80346987ea2" containerID="22dc87eee689b36fca6c865f4a31ecc77586d904100af1d369fe2ec5756f622c" exitCode=255 Apr 21 17:36:00.432748 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.432672 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" event={"ID":"75263093-7984-4bc3-b1d9-b80346987ea2","Type":"ContainerDied","Data":"22dc87eee689b36fca6c865f4a31ecc77586d904100af1d369fe2ec5756f622c"} Apr 21 17:36:00.432933 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.432920 2578 scope.go:117] "RemoveContainer" containerID="22dc87eee689b36fca6c865f4a31ecc77586d904100af1d369fe2ec5756f622c" Apr 21 17:36:00.785144 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:00.785037 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:36:00.785295 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:36:00.785192 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Apr 21 17:36:00.785295 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:36:00.785260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls podName:033302de-ab2b-4b50-8bf5-bea5f1870198 nodeName:}" failed. No retries permitted until 2026-04-21 17:36:04.785244402 +0000 UTC m=+168.507359355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tz8cg" (UID: "033302de-ab2b-4b50-8bf5-bea5f1870198") : secret "insights-runtime-extractor-tls" not found Apr 21 17:36:01.439650 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:01.439614 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" event={"ID":"7a543a5c-3a96-425b-8771-bc2371809c86","Type":"ContainerStarted","Data":"f6eadc79ae1a0263d44f97b2fcf99ba28d2d9b6e0c62d9c75fc82149c2d1a479"} Apr 21 17:36:01.440094 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:01.439921 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:36:01.440714 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:01.440695 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6788d84d6d-q6d2t" Apr 21 17:36:01.441268 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:01.441243 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-546698bbdf-pl4m4" event={"ID":"75263093-7984-4bc3-b1d9-b80346987ea2","Type":"ContainerStarted","Data":"0bb9b4af7bfc563bfec59d71a75657b249bb7eca78c3113b52d826523afd3def"} Apr 21 17:36:02.797424 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:02.797395 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:36:04.798043 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:04.798009 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:36:04.818804 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:04.818775 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:36:04.820943 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:04.820925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/033302de-ab2b-4b50-8bf5-bea5f1870198-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tz8cg\" (UID: \"033302de-ab2b-4b50-8bf5-bea5f1870198\") " pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:36:05.119820 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:05.119790 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tz8cg" Apr 21 17:36:05.232344 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:05.232307 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tz8cg"] Apr 21 17:36:05.235488 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:36:05.235459 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033302de_ab2b_4b50_8bf5_bea5f1870198.slice/crio-bd96308fdd65c90661e0a20bad3fc4cdfeca10c3a5450925fc1d3a9ae9b83b26 WatchSource:0}: Error finding container bd96308fdd65c90661e0a20bad3fc4cdfeca10c3a5450925fc1d3a9ae9b83b26: Status 404 returned error can't find the container with id bd96308fdd65c90661e0a20bad3fc4cdfeca10c3a5450925fc1d3a9ae9b83b26 Apr 21 17:36:05.451312 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:05.451221 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz8cg" event={"ID":"033302de-ab2b-4b50-8bf5-bea5f1870198","Type":"ContainerStarted","Data":"73fa0c0eb827d3d5dd3aec623a1f1c59128b478c991a6b650fb0ccbfc4ac5dba"} Apr 21 17:36:05.451312 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:05.451259 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz8cg" event={"ID":"033302de-ab2b-4b50-8bf5-bea5f1870198","Type":"ContainerStarted","Data":"bd96308fdd65c90661e0a20bad3fc4cdfeca10c3a5450925fc1d3a9ae9b83b26"} Apr 21 17:36:06.455715 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:06.455680 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz8cg" event={"ID":"033302de-ab2b-4b50-8bf5-bea5f1870198","Type":"ContainerStarted","Data":"79ca4e9fcc25514b9a7b60b04ebc666e2bc5702d1a54f7c64c1b739002532dd8"} Apr 21 17:36:07.460699 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:07.460612 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tz8cg" event={"ID":"033302de-ab2b-4b50-8bf5-bea5f1870198","Type":"ContainerStarted","Data":"62b90005d085affe27954bb50a30674355464ded54d73316bc2a0da2b1151468"} Apr 21 17:36:07.478132 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:07.478086 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tz8cg" podStartSLOduration=9.559325307 podStartE2EDuration="11.47807394s" podCreationTimestamp="2026-04-21 17:35:56 +0000 UTC" firstStartedPulling="2026-04-21 17:36:05.290003419 +0000 UTC m=+169.012118371" lastFinishedPulling="2026-04-21 17:36:07.208752048 +0000 UTC m=+170.930867004" observedRunningTime="2026-04-21 17:36:07.477464949 +0000 UTC m=+171.199579924" watchObservedRunningTime="2026-04-21 17:36:07.47807394 +0000 UTC m=+171.200188914" Apr 21 17:36:26.469926 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.469895 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-trhjt"] Apr 21 17:36:26.474761 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.474745 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.477620 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.477598 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Apr 21 17:36:26.477620 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.477609 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Apr 21 17:36:26.477786 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.477643 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Apr 21 17:36:26.477786 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.477600 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Apr 21 17:36:26.477786 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.477599 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Apr 21 17:36:26.478264 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.478248 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Apr 21 17:36:26.478648 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.478635 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xm58w\"" Apr 21 17:36:26.573245 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-tls\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573245 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbgh\" (UniqueName: \"kubernetes.io/projected/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-kube-api-access-4hbgh\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573502 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573502 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-metrics-client-ca\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573502 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-wtmp\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573502 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-textfile\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573502 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573689 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-sys\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.573689 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.573601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-root\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674023 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.673995 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674194 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674031 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-metrics-client-ca\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674194 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674063 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-wtmp\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674194 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674168 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-textfile\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674194 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-wtmp\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674211 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674252 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-sys\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674289 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-root\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674342 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-tls\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674367 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbgh\" (UniqueName: \"kubernetes.io/projected/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-kube-api-access-4hbgh\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674385 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-sys\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674720 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674528 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-textfile\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674779 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674711 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-root\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674779 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674763 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-metrics-client-ca\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.674879 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.674831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.676609 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.676590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.676675 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.676596 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-node-exporter-tls\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.691406 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.691383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbgh\" (UniqueName: \"kubernetes.io/projected/8e7411a8-d5bb-4c7d-9d23-7785d9751b6a-kube-api-access-4hbgh\") pod \"node-exporter-trhjt\" (UID: \"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a\") " pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.783577 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:26.783497 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-trhjt" Apr 21 17:36:26.791274 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:36:26.791249 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7411a8_d5bb_4c7d_9d23_7785d9751b6a.slice/crio-94c098b703953ebd90310c8f9b1669a8e9e67f389d3f06051fb52e84fdf278ef WatchSource:0}: Error finding container 94c098b703953ebd90310c8f9b1669a8e9e67f389d3f06051fb52e84fdf278ef: Status 404 returned error can't find the container with id 94c098b703953ebd90310c8f9b1669a8e9e67f389d3f06051fb52e84fdf278ef Apr 21 17:36:27.506701 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:27.506665 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trhjt" event={"ID":"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a","Type":"ContainerStarted","Data":"94c098b703953ebd90310c8f9b1669a8e9e67f389d3f06051fb52e84fdf278ef"} Apr 21 17:36:28.510525 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:28.510493 2578 generic.go:358] "Generic (PLEG): container finished" podID="8e7411a8-d5bb-4c7d-9d23-7785d9751b6a" containerID="dc157bb4ce382066ead54e56cbb81ce8bcc009103f1dc764721ef95910437212" exitCode=0 Apr 21 17:36:28.510903 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:28.510566 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trhjt" event={"ID":"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a","Type":"ContainerDied","Data":"dc157bb4ce382066ead54e56cbb81ce8bcc009103f1dc764721ef95910437212"} Apr 21 17:36:29.514428 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:29.514393 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trhjt" event={"ID":"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a","Type":"ContainerStarted","Data":"54613f6a61bdde2c1baba8e50717fa21c8ce5a364346f51f70b1e237d08a528f"} Apr 21 17:36:29.514791 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:29.514445 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-trhjt" event={"ID":"8e7411a8-d5bb-4c7d-9d23-7785d9751b6a","Type":"ContainerStarted","Data":"51f1ffe0edf5c285dc47be837ca69e195307ffbbddfee0b2e31a9ff41d0d8dec"} Apr 21 17:36:29.536605 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:29.536543 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-trhjt" podStartSLOduration=2.8365825940000002 podStartE2EDuration="3.536528387s" podCreationTimestamp="2026-04-21 17:36:26 +0000 UTC" firstStartedPulling="2026-04-21 17:36:26.793128652 +0000 UTC m=+190.515243607" lastFinishedPulling="2026-04-21 17:36:27.493074448 +0000 UTC m=+191.215189400" observedRunningTime="2026-04-21 17:36:29.534866083 +0000 UTC m=+193.256981056" watchObservedRunningTime="2026-04-21 17:36:29.536528387 +0000 UTC m=+193.258643427" Apr 21 17:36:39.100263 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:39.100224 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" podUID="413bc0c4-35c9-4de3-8aff-47a10cfd1322" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Apr 21 17:36:41.004120 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.004086 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f84d86b-5wqs4"] Apr 21 17:36:41.004571 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:36:41.004316 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" podUID="b1b62c08-2675-4759-887b-5dd01667868c" Apr 21 17:36:41.545872 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.545841 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:36:41.549859 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.549839 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:36:41.590929 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.590902 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-registry-certificates\") pod \"b1b62c08-2675-4759-887b-5dd01667868c\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " Apr 21 17:36:41.591019 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.590944 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-installation-pull-secrets\") pod \"b1b62c08-2675-4759-887b-5dd01667868c\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " Apr 21 17:36:41.591019 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.590966 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-bound-sa-token\") pod \"b1b62c08-2675-4759-887b-5dd01667868c\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " Apr 21 17:36:41.591019 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.590996 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b62c08-2675-4759-887b-5dd01667868c-ca-trust-extracted\") pod \"b1b62c08-2675-4759-887b-5dd01667868c\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " Apr 21 17:36:41.591133 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.591034 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-image-registry-private-configuration\") pod \"b1b62c08-2675-4759-887b-5dd01667868c\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " Apr 21 17:36:41.591133 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.591057 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r22hr\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-kube-api-access-r22hr\") pod \"b1b62c08-2675-4759-887b-5dd01667868c\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " Apr 21 17:36:41.591133 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.591100 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-trusted-ca\") pod \"b1b62c08-2675-4759-887b-5dd01667868c\" (UID: \"b1b62c08-2675-4759-887b-5dd01667868c\") " Apr 21 17:36:41.591380 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.591295 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b1b62c08-2675-4759-887b-5dd01667868c" (UID: "b1b62c08-2675-4759-887b-5dd01667868c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 21 17:36:41.591531 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.591385 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b62c08-2675-4759-887b-5dd01667868c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b1b62c08-2675-4759-887b-5dd01667868c" (UID: "b1b62c08-2675-4759-887b-5dd01667868c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 21 17:36:41.591639 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.591611 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b1b62c08-2675-4759-887b-5dd01667868c" (UID: "b1b62c08-2675-4759-887b-5dd01667868c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 21 17:36:41.593381 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.593356 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b1b62c08-2675-4759-887b-5dd01667868c" (UID: "b1b62c08-2675-4759-887b-5dd01667868c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:36:41.593501 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.593473 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-kube-api-access-r22hr" (OuterVolumeSpecName: "kube-api-access-r22hr") pod "b1b62c08-2675-4759-887b-5dd01667868c" (UID: "b1b62c08-2675-4759-887b-5dd01667868c"). InnerVolumeSpecName "kube-api-access-r22hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:36:41.593621 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.593606 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b1b62c08-2675-4759-887b-5dd01667868c" (UID: "b1b62c08-2675-4759-887b-5dd01667868c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 21 17:36:41.593695 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.593669 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b1b62c08-2675-4759-887b-5dd01667868c" (UID: "b1b62c08-2675-4759-887b-5dd01667868c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 21 17:36:41.692723 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.692684 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-bound-sa-token\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:41.692723 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.692713 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b62c08-2675-4759-887b-5dd01667868c-ca-trust-extracted\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:41.692723 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.692725 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-image-registry-private-configuration\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:41.692953 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.692735 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r22hr\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-kube-api-access-r22hr\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:41.692953 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.692745 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-trusted-ca\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:41.692953 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.692754 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b62c08-2675-4759-887b-5dd01667868c-registry-certificates\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:41.692953 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:41.692764 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b62c08-2675-4759-887b-5dd01667868c-installation-pull-secrets\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:42.547716 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:42.547679 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f84d86b-5wqs4" Apr 21 17:36:42.583662 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:42.583637 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f84d86b-5wqs4"] Apr 21 17:36:42.587130 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:42.587110 2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68f84d86b-5wqs4"] Apr 21 17:36:42.700781 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:42.700740 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b62c08-2675-4759-887b-5dd01667868c-registry-tls\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:36:42.800812 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:42.800732 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b62c08-2675-4759-887b-5dd01667868c" path="/var/lib/kubelet/pods/b1b62c08-2675-4759-887b-5dd01667868c/volumes" Apr 21 17:36:49.100120 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:49.100081 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" podUID="413bc0c4-35c9-4de3-8aff-47a10cfd1322" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Apr 21 17:36:59.099899 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:59.099859 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" podUID="413bc0c4-35c9-4de3-8aff-47a10cfd1322" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Apr 21 17:36:59.100276 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:59.099941 2578 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" Apr 21 17:36:59.100432 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:59.100382 2578 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"58fdcfd3fbcba8eeca21e850d6ab028435cd068b5892aa44e77a7f6a51b282ff"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" containerMessage="Container service-proxy failed liveness probe, will be restarted" Apr 21 17:36:59.100483 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:59.100455 2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" podUID="413bc0c4-35c9-4de3-8aff-47a10cfd1322" containerName="service-proxy" containerID="cri-o://58fdcfd3fbcba8eeca21e850d6ab028435cd068b5892aa44e77a7f6a51b282ff" gracePeriod=30 Apr 21 17:36:59.594006 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:59.593976 2578 generic.go:358] "Generic (PLEG): container finished" podID="413bc0c4-35c9-4de3-8aff-47a10cfd1322" containerID="58fdcfd3fbcba8eeca21e850d6ab028435cd068b5892aa44e77a7f6a51b282ff" exitCode=2 Apr 21 17:36:59.594178 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:59.594048 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" event={"ID":"413bc0c4-35c9-4de3-8aff-47a10cfd1322","Type":"ContainerDied","Data":"58fdcfd3fbcba8eeca21e850d6ab028435cd068b5892aa44e77a7f6a51b282ff"} Apr 21 17:36:59.594178 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:36:59.594085 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdc6878b7-x9rkj" event={"ID":"413bc0c4-35c9-4de3-8aff-47a10cfd1322","Type":"ContainerStarted","Data":"6f27e39ed144102698a1d25c7046e2092cdcc632d97b56ac4f87fa04b3bc9182"} Apr 21 17:37:27.736795 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:27.736756 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:37:27.738988 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:27.738965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb61ff1-0bdf-4220-99ee-6f860546dc41-metrics-certs\") pod \"network-metrics-daemon-z46hp\" (UID: \"cbb61ff1-0bdf-4220-99ee-6f860546dc41\") " pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:37:28.001766 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:28.001673 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pqc26\"" Apr 21 17:37:28.009154 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:28.009133 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z46hp" Apr 21 17:37:28.134625 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:28.134593 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z46hp"] Apr 21 17:37:28.139772 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:37:28.137934 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb61ff1_0bdf_4220_99ee_6f860546dc41.slice/crio-9a00dadd5ab93792a34872c304fbbc192c945857330207b61316cd92ca8d908c WatchSource:0}: Error finding container 9a00dadd5ab93792a34872c304fbbc192c945857330207b61316cd92ca8d908c: Status 404 returned error can't find the container with id 9a00dadd5ab93792a34872c304fbbc192c945857330207b61316cd92ca8d908c Apr 21 17:37:28.663978 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:28.663940 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z46hp" event={"ID":"cbb61ff1-0bdf-4220-99ee-6f860546dc41","Type":"ContainerStarted","Data":"9a00dadd5ab93792a34872c304fbbc192c945857330207b61316cd92ca8d908c"} Apr 21 17:37:29.668734 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:29.668696 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z46hp" event={"ID":"cbb61ff1-0bdf-4220-99ee-6f860546dc41","Type":"ContainerStarted","Data":"6f14e248d4da8de6ec476de0abe83f86563c98f86e14ce67d464f7c8600baa7d"} Apr 21 17:37:29.669128 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:29.668741 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z46hp" event={"ID":"cbb61ff1-0bdf-4220-99ee-6f860546dc41","Type":"ContainerStarted","Data":"865c52e817eb9625fabba5532555d875905e6b7558c4bb6d433f7c8e435ca124"} Apr 21 17:37:29.686836 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:29.686512 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z46hp" podStartSLOduration=252.793395499 podStartE2EDuration="4m13.68649124s" podCreationTimestamp="2026-04-21 17:33:16 +0000 UTC" firstStartedPulling="2026-04-21 17:37:28.142096877 +0000 UTC m=+251.864211829" lastFinishedPulling="2026-04-21 17:37:29.035192618 +0000 UTC m=+252.757307570" observedRunningTime="2026-04-21 17:37:29.685510125 +0000 UTC m=+253.407625102" watchObservedRunningTime="2026-04-21 17:37:29.68649124 +0000 UTC m=+253.408606216" Apr 21 17:37:55.412860 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:37:55.412812 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" podUID="dece6707-af2b-4518-b0f4-95ec5f993643" Apr 21 17:37:55.412860 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:37:55.412824 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4qz45" podUID="137fa6ca-d790-417d-bc67-5a19c38af051" Apr 21 17:37:55.735801 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:55.735730 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:37:55.735966 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:55.735735 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qz45" Apr 21 17:37:58.762364 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:58.762330 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:37:58.764628 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:58.764610 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dece6707-af2b-4518-b0f4-95ec5f993643-networking-console-plugin-cert\") pod \"networking-console-plugin-cb95c66f6-kzkr7\" (UID: \"dece6707-af2b-4518-b0f4-95ec5f993643\") " pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:37:58.863029 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:58.862990 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:37:58.863029 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:58.863033 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:37:58.865205 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:58.865177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/137fa6ca-d790-417d-bc67-5a19c38af051-metrics-tls\") pod \"dns-default-4qz45\" (UID: \"137fa6ca-d790-417d-bc67-5a19c38af051\") " pod="openshift-dns/dns-default-4qz45" Apr 21 17:37:58.865332 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:58.865315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b8ca466-7111-427f-891f-efc7ebe9d92d-cert\") pod \"ingress-canary-8bv2c\" (UID: \"1b8ca466-7111-427f-891f-efc7ebe9d92d\") " pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:37:59.039843 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.039764 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hv5zp\"" Apr 21 17:37:59.040752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.040737 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mlvk8\"" Apr 21 17:37:59.046767 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.046748 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" Apr 21 17:37:59.046863 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.046778 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4qz45" Apr 21 17:37:59.102743 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.102714 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4fbh6\"" Apr 21 17:37:59.110168 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.109725 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8bv2c" Apr 21 17:37:59.186243 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.186217 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4qz45"] Apr 21 17:37:59.188251 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:37:59.188222 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137fa6ca_d790_417d_bc67_5a19c38af051.slice/crio-0b1d2a968ce7d7d44debe93e663857e7718112aba11488f09037baf3ec3515cc WatchSource:0}: Error finding container 0b1d2a968ce7d7d44debe93e663857e7718112aba11488f09037baf3ec3515cc: Status 404 returned error can't find the container with id 0b1d2a968ce7d7d44debe93e663857e7718112aba11488f09037baf3ec3515cc Apr 21 17:37:59.203013 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.202985 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7"] Apr 21 17:37:59.205582 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:37:59.205550 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddece6707_af2b_4518_b0f4_95ec5f993643.slice/crio-faf3c477eb30137ee07a251bb182bfd5bf074c0cc66eed458b5eb5542f260b1f WatchSource:0}: Error finding container faf3c477eb30137ee07a251bb182bfd5bf074c0cc66eed458b5eb5542f260b1f: Status 404 returned error can't find the container with id faf3c477eb30137ee07a251bb182bfd5bf074c0cc66eed458b5eb5542f260b1f Apr 21 17:37:59.244292 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.244271 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8bv2c"] Apr 21 17:37:59.246527 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:37:59.246501 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8ca466_7111_427f_891f_efc7ebe9d92d.slice/crio-47f1185870f29f0db61916cf77b595373006fde2fb322baa0aac14f99e15cbf5 WatchSource:0}: Error finding container 47f1185870f29f0db61916cf77b595373006fde2fb322baa0aac14f99e15cbf5: Status 404 returned error can't find the container with id 47f1185870f29f0db61916cf77b595373006fde2fb322baa0aac14f99e15cbf5 Apr 21 17:37:59.748600 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.748538 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" event={"ID":"dece6707-af2b-4518-b0f4-95ec5f993643","Type":"ContainerStarted","Data":"faf3c477eb30137ee07a251bb182bfd5bf074c0cc66eed458b5eb5542f260b1f"} Apr 21 17:37:59.750209 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.750167 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bv2c" event={"ID":"1b8ca466-7111-427f-891f-efc7ebe9d92d","Type":"ContainerStarted","Data":"47f1185870f29f0db61916cf77b595373006fde2fb322baa0aac14f99e15cbf5"} Apr 21 17:37:59.751353 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:37:59.751305 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qz45" event={"ID":"137fa6ca-d790-417d-bc67-5a19c38af051","Type":"ContainerStarted","Data":"0b1d2a968ce7d7d44debe93e663857e7718112aba11488f09037baf3ec3515cc"} Apr 21 17:38:01.758928 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.758835 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" event={"ID":"dece6707-af2b-4518-b0f4-95ec5f993643","Type":"ContainerStarted","Data":"71f0d6befee8dc8216bea574cc6577592fcc5e59758afe68c86e81baf86a2685"} Apr 21 17:38:01.760224 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.760199 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8bv2c" event={"ID":"1b8ca466-7111-427f-891f-efc7ebe9d92d","Type":"ContainerStarted","Data":"72165c73512339fcb2c840955ec74954a7fd765c88775e9a6b8f626ba5818376"} Apr 21 17:38:01.761580 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.761558 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qz45" event={"ID":"137fa6ca-d790-417d-bc67-5a19c38af051","Type":"ContainerStarted","Data":"6f7b5b27739f5b43c1ff1f1e94e9d90dce2ea37c99e11edac8cddcfb53ecae76"} Apr 21 17:38:01.761670 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.761585 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4qz45" event={"ID":"137fa6ca-d790-417d-bc67-5a19c38af051","Type":"ContainerStarted","Data":"5b85fe89e08bf5b20edc8f8b3f65e1e2649b23e0eee781f02b107d3aee85c285"} Apr 21 17:38:01.761706 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.761682 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4qz45" Apr 21 17:38:01.779934 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.779886 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-cb95c66f6-kzkr7" podStartSLOduration=265.855833365 podStartE2EDuration="4m27.779869924s" podCreationTimestamp="2026-04-21 17:33:34 +0000 UTC" firstStartedPulling="2026-04-21 17:37:59.207719168 +0000 UTC m=+282.929834121" lastFinishedPulling="2026-04-21 17:38:01.131755726 +0000 UTC m=+284.853870680" observedRunningTime="2026-04-21 17:38:01.779525859 +0000 UTC m=+285.501640835" watchObservedRunningTime="2026-04-21 17:38:01.779869924 +0000 UTC m=+285.501984898" Apr 21 17:38:01.810765 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.810711 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-4qz45" podStartSLOduration=251.86818223 podStartE2EDuration="4m13.810693142s" podCreationTimestamp="2026-04-21 17:33:48 +0000 UTC" firstStartedPulling="2026-04-21 17:37:59.18999413 +0000 UTC m=+282.912109081" lastFinishedPulling="2026-04-21 17:38:01.132505027 +0000 UTC m=+284.854619993" observedRunningTime="2026-04-21 17:38:01.810193997 +0000 UTC m=+285.532308971" watchObservedRunningTime="2026-04-21 17:38:01.810693142 +0000 UTC m=+285.532808118" Apr 21 17:38:01.837668 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:01.837627 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8bv2c" podStartSLOduration=251.949128823 podStartE2EDuration="4m13.83761449s" podCreationTimestamp="2026-04-21 17:33:48 +0000 UTC" firstStartedPulling="2026-04-21 17:37:59.248282577 +0000 UTC m=+282.970397533" lastFinishedPulling="2026-04-21 17:38:01.136768241 +0000 UTC m=+284.858883200" observedRunningTime="2026-04-21 17:38:01.835667903 +0000 UTC m=+285.557782877" watchObservedRunningTime="2026-04-21 17:38:01.83761449 +0000 UTC m=+285.559729463" Apr 21 17:38:11.766877 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:11.766845 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4qz45" Apr 21 17:38:16.713625 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:16.713588 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:38:16.714033 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:16.713598 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:38:16.720479 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:38:16.720456 2578 kubelet.go:1628] "Image garbage collection succeeded" Apr 21 17:40:59.232700 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.232657 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb"] Apr 21 17:40:59.235581 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.235566 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.238485 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.238463 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Apr 21 17:40:59.238750 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.238734 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Apr 21 17:40:59.239988 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.239972 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-254cv\"" Apr 21 17:40:59.248472 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.248352 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb"] Apr 21 17:40:59.357797 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.357758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8add952f-3146-4786-a24a-2e0e8f70e91c-tmp\") pod \"openshift-lws-operator-bfc7f696d-qtxrb\" (UID: \"8add952f-3146-4786-a24a-2e0e8f70e91c\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.357982 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.357838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t862n\" (UniqueName: \"kubernetes.io/projected/8add952f-3146-4786-a24a-2e0e8f70e91c-kube-api-access-t862n\") pod \"openshift-lws-operator-bfc7f696d-qtxrb\" (UID: \"8add952f-3146-4786-a24a-2e0e8f70e91c\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.458683 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.458640 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t862n\" (UniqueName: \"kubernetes.io/projected/8add952f-3146-4786-a24a-2e0e8f70e91c-kube-api-access-t862n\") pod \"openshift-lws-operator-bfc7f696d-qtxrb\" (UID: \"8add952f-3146-4786-a24a-2e0e8f70e91c\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.458683 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.458690 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8add952f-3146-4786-a24a-2e0e8f70e91c-tmp\") pod \"openshift-lws-operator-bfc7f696d-qtxrb\" (UID: \"8add952f-3146-4786-a24a-2e0e8f70e91c\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.459018 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.458993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8add952f-3146-4786-a24a-2e0e8f70e91c-tmp\") pod \"openshift-lws-operator-bfc7f696d-qtxrb\" (UID: \"8add952f-3146-4786-a24a-2e0e8f70e91c\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.467337 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.467307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t862n\" (UniqueName: \"kubernetes.io/projected/8add952f-3146-4786-a24a-2e0e8f70e91c-kube-api-access-t862n\") pod \"openshift-lws-operator-bfc7f696d-qtxrb\" (UID: \"8add952f-3146-4786-a24a-2e0e8f70e91c\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.544902 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.544835 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" Apr 21 17:40:59.665870 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.665842 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb"] Apr 21 17:40:59.668836 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:40:59.668807 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8add952f_3146_4786_a24a_2e0e8f70e91c.slice/crio-62e77b46c49206c9ab08ba65d81e6db3645e36d1437c09d1a544ad4a18cafd1b WatchSource:0}: Error finding container 62e77b46c49206c9ab08ba65d81e6db3645e36d1437c09d1a544ad4a18cafd1b: Status 404 returned error can't find the container with id 62e77b46c49206c9ab08ba65d81e6db3645e36d1437c09d1a544ad4a18cafd1b Apr 21 17:40:59.670186 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:40:59.670166 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 21 17:41:00.214794 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:00.214762 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" event={"ID":"8add952f-3146-4786-a24a-2e0e8f70e91c","Type":"ContainerStarted","Data":"62e77b46c49206c9ab08ba65d81e6db3645e36d1437c09d1a544ad4a18cafd1b"} Apr 21 17:41:02.222746 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:02.222711 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" event={"ID":"8add952f-3146-4786-a24a-2e0e8f70e91c","Type":"ContainerStarted","Data":"ee4dfd09d724d05f57bdbafd38af74d5076e60c06db975705f5d56099280272d"} Apr 21 17:41:02.238827 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:02.238776 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-qtxrb" podStartSLOduration=0.78047119 podStartE2EDuration="3.238757889s" podCreationTimestamp="2026-04-21 17:40:59 +0000 UTC" firstStartedPulling="2026-04-21 17:40:59.670288679 +0000 UTC m=+463.392403632" lastFinishedPulling="2026-04-21 17:41:02.128575375 +0000 UTC m=+465.850690331" observedRunningTime="2026-04-21 17:41:02.237255834 +0000 UTC m=+465.959370808" watchObservedRunningTime="2026-04-21 17:41:02.238757889 +0000 UTC m=+465.960872864" Apr 21 17:41:19.099040 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.099003 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz"] Apr 21 17:41:19.102461 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.102405 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.107391 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.107367 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Apr 21 17:41:19.107391 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.107383 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Apr 21 17:41:19.107601 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.107582 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Apr 21 17:41:19.107683 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.107663 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gf9sq\"" Apr 21 17:41:19.107740 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.107720 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Apr 21 17:41:19.127864 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.127840 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz"] Apr 21 17:41:19.194719 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.194685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ba106c97-f27e-40e0-aa44-9c2a95887924-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.194841 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.194726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z29mk\" (UniqueName: \"kubernetes.io/projected/ba106c97-f27e-40e0-aa44-9c2a95887924-kube-api-access-z29mk\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.194841 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.194789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba106c97-f27e-40e0-aa44-9c2a95887924-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.295356 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.295328 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba106c97-f27e-40e0-aa44-9c2a95887924-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.295485 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.295385 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ba106c97-f27e-40e0-aa44-9c2a95887924-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.295485 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.295438 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z29mk\" (UniqueName: \"kubernetes.io/projected/ba106c97-f27e-40e0-aa44-9c2a95887924-kube-api-access-z29mk\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.297845 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.297813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ba106c97-f27e-40e0-aa44-9c2a95887924-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.297943 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.297818 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ba106c97-f27e-40e0-aa44-9c2a95887924-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.306831 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.306807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z29mk\" (UniqueName: \"kubernetes.io/projected/ba106c97-f27e-40e0-aa44-9c2a95887924-kube-api-access-z29mk\") pod \"opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz\" (UID: \"ba106c97-f27e-40e0-aa44-9c2a95887924\") " pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.411989 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.411957 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:19.538334 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:19.538213 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz"] Apr 21 17:41:19.540980 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:41:19.540951 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba106c97_f27e_40e0_aa44_9c2a95887924.slice/crio-c5f5b3df67857a96f0545e13054232a6e22ffba8b958e2fa246bb2a9dd4a34e5 WatchSource:0}: Error finding container c5f5b3df67857a96f0545e13054232a6e22ffba8b958e2fa246bb2a9dd4a34e5: Status 404 returned error can't find the container with id c5f5b3df67857a96f0545e13054232a6e22ffba8b958e2fa246bb2a9dd4a34e5 Apr 21 17:41:20.270429 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:20.270391 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" event={"ID":"ba106c97-f27e-40e0-aa44-9c2a95887924","Type":"ContainerStarted","Data":"c5f5b3df67857a96f0545e13054232a6e22ffba8b958e2fa246bb2a9dd4a34e5"} Apr 21 17:41:22.281618 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:22.281582 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" event={"ID":"ba106c97-f27e-40e0-aa44-9c2a95887924","Type":"ContainerStarted","Data":"baedf64c773362c76f1a3a0f61cbfd8ac495ba5defd11f11344e921cf5c6f120"} Apr 21 17:41:22.282002 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:22.281641 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:22.304743 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:22.304694 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" podStartSLOduration=0.726964669 podStartE2EDuration="3.304678266s" podCreationTimestamp="2026-04-21 17:41:19 +0000 UTC" firstStartedPulling="2026-04-21 17:41:19.542593238 +0000 UTC m=+483.264708190" lastFinishedPulling="2026-04-21 17:41:22.120306835 +0000 UTC m=+485.842421787" observedRunningTime="2026-04-21 17:41:22.302864574 +0000 UTC m=+486.024979548" watchObservedRunningTime="2026-04-21 17:41:22.304678266 +0000 UTC m=+486.026793239" Apr 21 17:41:33.286843 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:33.286815 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz" Apr 21 17:41:37.551252 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.551218 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv"] Apr 21 17:41:37.639043 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.639009 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv"] Apr 21 17:41:37.639196 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.639148 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.642543 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.642519 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Apr 21 17:41:37.642831 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.642810 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nq62s\"" Apr 21 17:41:37.643015 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.642983 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Apr 21 17:41:37.643113 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.643024 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Apr 21 17:41:37.644709 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.644687 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Apr 21 17:41:37.825349 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.825263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfbw9\" (UniqueName: \"kubernetes.io/projected/95b197dd-ee42-4148-9125-dcf0bd52abf1-kube-api-access-mfbw9\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.825349 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.825300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95b197dd-ee42-4148-9125-dcf0bd52abf1-tmp\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.825349 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.825329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b197dd-ee42-4148-9125-dcf0bd52abf1-tls-certs\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.926705 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.926667 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mfbw9\" (UniqueName: \"kubernetes.io/projected/95b197dd-ee42-4148-9125-dcf0bd52abf1-kube-api-access-mfbw9\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.926705 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.926708 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95b197dd-ee42-4148-9125-dcf0bd52abf1-tmp\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.926913 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.926889 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b197dd-ee42-4148-9125-dcf0bd52abf1-tls-certs\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.929015 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.928992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95b197dd-ee42-4148-9125-dcf0bd52abf1-tmp\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.929323 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.929303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b197dd-ee42-4148-9125-dcf0bd52abf1-tls-certs\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.948851 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.948827 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfbw9\" (UniqueName: \"kubernetes.io/projected/95b197dd-ee42-4148-9125-dcf0bd52abf1-kube-api-access-mfbw9\") pod \"kube-auth-proxy-548b8d8fcb-z2ztv\" (UID: \"95b197dd-ee42-4148-9125-dcf0bd52abf1\") " pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:37.951881 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:37.951859 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" Apr 21 17:41:38.108734 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:38.108704 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv"] Apr 21 17:41:38.115182 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:41:38.115150 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b197dd_ee42_4148_9125_dcf0bd52abf1.slice/crio-8ec58e4b5836fa9e0eeffc21dea63ae12480899ca153b8deff2fd7950c0023bd WatchSource:0}: Error finding container 8ec58e4b5836fa9e0eeffc21dea63ae12480899ca153b8deff2fd7950c0023bd: Status 404 returned error can't find the container with id 8ec58e4b5836fa9e0eeffc21dea63ae12480899ca153b8deff2fd7950c0023bd Apr 21 17:41:38.324047 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:38.324016 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" event={"ID":"95b197dd-ee42-4148-9125-dcf0bd52abf1","Type":"ContainerStarted","Data":"8ec58e4b5836fa9e0eeffc21dea63ae12480899ca153b8deff2fd7950c0023bd"} Apr 21 17:41:42.337734 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.337697 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" event={"ID":"95b197dd-ee42-4148-9125-dcf0bd52abf1","Type":"ContainerStarted","Data":"c5e3b87b8435539f07f03ea883681779f79529e23515349e41b00ffabd0ebc2a"} Apr 21 17:41:42.381168 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.381120 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-548b8d8fcb-z2ztv" podStartSLOduration=1.353387039 podStartE2EDuration="5.381104912s" podCreationTimestamp="2026-04-21 17:41:37 +0000 UTC" firstStartedPulling="2026-04-21 17:41:38.116836308 +0000 UTC m=+501.838951261" lastFinishedPulling="2026-04-21 17:41:42.144554179 +0000 UTC m=+505.866669134" observedRunningTime="2026-04-21 17:41:42.379935593 +0000 UTC m=+506.102050581" watchObservedRunningTime="2026-04-21 17:41:42.381104912 +0000 UTC m=+506.103219890" Apr 21 17:41:42.580820 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.580738 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-858dbf95b8-2ptgp"] Apr 21 17:41:42.583143 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.583128 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:42.585907 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.585884 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Apr 21 17:41:42.586050 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.585884 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-f66k2\"" Apr 21 17:41:42.603847 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.603824 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-858dbf95b8-2ptgp"] Apr 21 17:41:42.667475 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.667445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:42.667636 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.667497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw26t\" (UniqueName: \"kubernetes.io/projected/fb301ae7-390b-4a44-93fb-a6b34d17a309-kube-api-access-fw26t\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:42.768650 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.768613 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:42.768815 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.768675 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fw26t\" (UniqueName: \"kubernetes.io/projected/fb301ae7-390b-4a44-93fb-a6b34d17a309-kube-api-access-fw26t\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:42.768815 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:42.768774 2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Apr 21 17:41:42.768884 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:42.768834 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert podName:fb301ae7-390b-4a44-93fb-a6b34d17a309 nodeName:}" failed. No retries permitted until 2026-04-21 17:41:43.268818771 +0000 UTC m=+506.990933723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert") pod "odh-model-controller-858dbf95b8-2ptgp" (UID: "fb301ae7-390b-4a44-93fb-a6b34d17a309") : secret "odh-model-controller-webhook-cert" not found Apr 21 17:41:42.780887 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:42.780860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw26t\" (UniqueName: \"kubernetes.io/projected/fb301ae7-390b-4a44-93fb-a6b34d17a309-kube-api-access-fw26t\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:43.271851 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:43.271810 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:43.272018 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:43.271950 2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Apr 21 17:41:43.272061 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:43.272024 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert podName:fb301ae7-390b-4a44-93fb-a6b34d17a309 nodeName:}" failed. No retries permitted until 2026-04-21 17:41:44.272006858 +0000 UTC m=+507.994121809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert") pod "odh-model-controller-858dbf95b8-2ptgp" (UID: "fb301ae7-390b-4a44-93fb-a6b34d17a309") : secret "odh-model-controller-webhook-cert" not found Apr 21 17:41:44.278513 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:44.278478 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:44.280870 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:44.280829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb301ae7-390b-4a44-93fb-a6b34d17a309-cert\") pod \"odh-model-controller-858dbf95b8-2ptgp\" (UID: \"fb301ae7-390b-4a44-93fb-a6b34d17a309\") " pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:44.392168 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:44.392138 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:44.519459 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:44.519437 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-858dbf95b8-2ptgp"] Apr 21 17:41:44.521708 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:41:44.521676 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb301ae7_390b_4a44_93fb_a6b34d17a309.slice/crio-e63fb1777669ee2d5079f5c849858f2a1e17c2af63981833fa8e85c88ff6dede WatchSource:0}: Error finding container e63fb1777669ee2d5079f5c849858f2a1e17c2af63981833fa8e85c88ff6dede: Status 404 returned error can't find the container with id e63fb1777669ee2d5079f5c849858f2a1e17c2af63981833fa8e85c88ff6dede Apr 21 17:41:45.348227 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:45.348174 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" event={"ID":"fb301ae7-390b-4a44-93fb-a6b34d17a309","Type":"ContainerStarted","Data":"e63fb1777669ee2d5079f5c849858f2a1e17c2af63981833fa8e85c88ff6dede"} Apr 21 17:41:48.358194 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.358161 2578 generic.go:358] "Generic (PLEG): container finished" podID="fb301ae7-390b-4a44-93fb-a6b34d17a309" containerID="c9664ae3f1a46d0706ad12057c4fe425221a43ec366ba3f2b7aa2f67bb310885" exitCode=1 Apr 21 17:41:48.358662 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.358221 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" event={"ID":"fb301ae7-390b-4a44-93fb-a6b34d17a309","Type":"ContainerDied","Data":"c9664ae3f1a46d0706ad12057c4fe425221a43ec366ba3f2b7aa2f67bb310885"} Apr 21 17:41:48.358662 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.358549 2578 scope.go:117] "RemoveContainer" containerID="c9664ae3f1a46d0706ad12057c4fe425221a43ec366ba3f2b7aa2f67bb310885" Apr 21 17:41:48.447986 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.447959 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-856948b99f-kxqxm"] Apr 21 17:41:48.451192 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.451169 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:48.454155 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.454123 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Apr 21 17:41:48.454288 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.454260 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l2whr\"" Apr 21 17:41:48.475582 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.475557 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-856948b99f-kxqxm"] Apr 21 17:41:48.506819 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.506789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d2fq\" (UniqueName: \"kubernetes.io/projected/65669328-4726-4eba-b1d2-88e5025ad1dd-kube-api-access-7d2fq\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:48.506930 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.506829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:48.608088 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.608059 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7d2fq\" (UniqueName: \"kubernetes.io/projected/65669328-4726-4eba-b1d2-88e5025ad1dd-kube-api-access-7d2fq\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:48.608231 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.608094 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:48.608231 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:48.608204 2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Apr 21 17:41:48.608320 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:48.608263 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert podName:65669328-4726-4eba-b1d2-88e5025ad1dd nodeName:}" failed. No retries permitted until 2026-04-21 17:41:49.108243634 +0000 UTC m=+512.830358585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert") pod "kserve-controller-manager-856948b99f-kxqxm" (UID: "65669328-4726-4eba-b1d2-88e5025ad1dd") : secret "kserve-webhook-server-cert" not found Apr 21 17:41:48.623796 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:48.623774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d2fq\" (UniqueName: \"kubernetes.io/projected/65669328-4726-4eba-b1d2-88e5025ad1dd-kube-api-access-7d2fq\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:49.111347 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:49.111317 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:49.111516 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:49.111425 2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Apr 21 17:41:49.111516 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:41:49.111487 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert podName:65669328-4726-4eba-b1d2-88e5025ad1dd nodeName:}" failed. No retries permitted until 2026-04-21 17:41:50.11147174 +0000 UTC m=+513.833586691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert") pod "kserve-controller-manager-856948b99f-kxqxm" (UID: "65669328-4726-4eba-b1d2-88e5025ad1dd") : secret "kserve-webhook-server-cert" not found Apr 21 17:41:49.363199 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:49.363110 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" event={"ID":"fb301ae7-390b-4a44-93fb-a6b34d17a309","Type":"ContainerStarted","Data":"f212088e78b4d1960359c91e7e318af52ebab1ee94fd2fc8b4afcbfb0d73f478"} Apr 21 17:41:49.363580 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:49.363202 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:41:49.381313 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:49.381260 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" podStartSLOduration=3.238864715 podStartE2EDuration="7.381242006s" podCreationTimestamp="2026-04-21 17:41:42 +0000 UTC" firstStartedPulling="2026-04-21 17:41:44.522970577 +0000 UTC m=+508.245085529" lastFinishedPulling="2026-04-21 17:41:48.665347867 +0000 UTC m=+512.387462820" observedRunningTime="2026-04-21 17:41:49.380307287 +0000 UTC m=+513.102422261" watchObservedRunningTime="2026-04-21 17:41:49.381242006 +0000 UTC m=+513.103356981" Apr 21 17:41:50.120064 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:50.120028 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:50.122333 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:50.122307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65669328-4726-4eba-b1d2-88e5025ad1dd-cert\") pod \"kserve-controller-manager-856948b99f-kxqxm\" (UID: \"65669328-4726-4eba-b1d2-88e5025ad1dd\") " pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:50.263426 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:50.263390 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:50.398951 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:50.398887 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-856948b99f-kxqxm"] Apr 21 17:41:50.401849 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:41:50.401822 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65669328_4726_4eba_b1d2_88e5025ad1dd.slice/crio-486510457e8114d423243465611716ec2550b5230c8299b58f91ab846ebcd400 WatchSource:0}: Error finding container 486510457e8114d423243465611716ec2550b5230c8299b58f91ab846ebcd400: Status 404 returned error can't find the container with id 486510457e8114d423243465611716ec2550b5230c8299b58f91ab846ebcd400 Apr 21 17:41:51.372324 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:51.372290 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" event={"ID":"65669328-4726-4eba-b1d2-88e5025ad1dd","Type":"ContainerStarted","Data":"486510457e8114d423243465611716ec2550b5230c8299b58f91ab846ebcd400"} Apr 21 17:41:53.380212 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:53.380186 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" event={"ID":"65669328-4726-4eba-b1d2-88e5025ad1dd","Type":"ContainerStarted","Data":"0118072d7954e21a3840ff36b99526891fccb978558b317e2e8521b13f6328e4"} Apr 21 17:41:53.380599 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:53.380331 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:41:53.399195 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:53.399148 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" podStartSLOduration=2.491623415 podStartE2EDuration="5.399134393s" podCreationTimestamp="2026-04-21 17:41:48 +0000 UTC" firstStartedPulling="2026-04-21 17:41:50.403105372 +0000 UTC m=+514.125220326" lastFinishedPulling="2026-04-21 17:41:53.310616348 +0000 UTC m=+517.032731304" observedRunningTime="2026-04-21 17:41:53.398348587 +0000 UTC m=+517.120463646" watchObservedRunningTime="2026-04-21 17:41:53.399134393 +0000 UTC m=+517.121249366" Apr 21 17:41:54.745107 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.745069 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-klhcw"] Apr 21 17:41:54.747600 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.747581 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:54.750552 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.750531 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-qjlpt\"" Apr 21 17:41:54.750667 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.750624 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\"" Apr 21 17:41:54.750723 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.750681 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\"" Apr 21 17:41:54.762764 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.762742 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-klhcw"] Apr 21 17:41:54.855883 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.855853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjz2\" (UniqueName: \"kubernetes.io/projected/be5c812a-458d-4c30-bc78-cb6cdf056f73-kube-api-access-cmjz2\") pod \"servicemesh-operator3-55f49c5f94-klhcw\" (UID: \"be5c812a-458d-4c30-bc78-cb6cdf056f73\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:54.856065 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.855910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/be5c812a-458d-4c30-bc78-cb6cdf056f73-operator-config\") pod \"servicemesh-operator3-55f49c5f94-klhcw\" (UID: \"be5c812a-458d-4c30-bc78-cb6cdf056f73\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:54.956992 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.956958 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjz2\" (UniqueName: \"kubernetes.io/projected/be5c812a-458d-4c30-bc78-cb6cdf056f73-kube-api-access-cmjz2\") pod \"servicemesh-operator3-55f49c5f94-klhcw\" (UID: \"be5c812a-458d-4c30-bc78-cb6cdf056f73\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:54.957147 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.957018 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/be5c812a-458d-4c30-bc78-cb6cdf056f73-operator-config\") pod \"servicemesh-operator3-55f49c5f94-klhcw\" (UID: \"be5c812a-458d-4c30-bc78-cb6cdf056f73\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:54.959777 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.959750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/be5c812a-458d-4c30-bc78-cb6cdf056f73-operator-config\") pod \"servicemesh-operator3-55f49c5f94-klhcw\" (UID: \"be5c812a-458d-4c30-bc78-cb6cdf056f73\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:54.968403 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:54.968378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjz2\" (UniqueName: \"kubernetes.io/projected/be5c812a-458d-4c30-bc78-cb6cdf056f73-kube-api-access-cmjz2\") pod \"servicemesh-operator3-55f49c5f94-klhcw\" (UID: \"be5c812a-458d-4c30-bc78-cb6cdf056f73\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:55.057129 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:55.057043 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:55.186912 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:55.186884 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-klhcw"] Apr 21 17:41:55.189848 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:41:55.189817 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5c812a_458d_4c30_bc78_cb6cdf056f73.slice/crio-bb50ab8ebc500c477c6549cb0edb9efdce7b5ace595b70ed31264727b4942b85 WatchSource:0}: Error finding container bb50ab8ebc500c477c6549cb0edb9efdce7b5ace595b70ed31264727b4942b85: Status 404 returned error can't find the container with id bb50ab8ebc500c477c6549cb0edb9efdce7b5ace595b70ed31264727b4942b85 Apr 21 17:41:55.389769 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:55.389733 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" event={"ID":"be5c812a-458d-4c30-bc78-cb6cdf056f73","Type":"ContainerStarted","Data":"bb50ab8ebc500c477c6549cb0edb9efdce7b5ace595b70ed31264727b4942b85"} Apr 21 17:41:59.403614 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:59.403531 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" event={"ID":"be5c812a-458d-4c30-bc78-cb6cdf056f73","Type":"ContainerStarted","Data":"f5af86636734106395055462034b114dc7602d5a4b6e4e82c552ce3f0a028d2e"} Apr 21 17:41:59.403614 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:59.403588 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:41:59.425523 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:41:59.425476 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" podStartSLOduration=2.0176331 podStartE2EDuration="5.425464088s" podCreationTimestamp="2026-04-21 17:41:54 +0000 UTC" firstStartedPulling="2026-04-21 17:41:55.192543614 +0000 UTC m=+518.914658570" lastFinishedPulling="2026-04-21 17:41:58.6003746 +0000 UTC m=+522.322489558" observedRunningTime="2026-04-21 17:41:59.424026319 +0000 UTC m=+523.146141294" watchObservedRunningTime="2026-04-21 17:41:59.425464088 +0000 UTC m=+523.147579062" Apr 21 17:42:00.369037 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:00.369003 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-858dbf95b8-2ptgp" Apr 21 17:42:02.282469 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.282437 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c"] Apr 21 17:42:02.284648 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.284619 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.289292 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.289266 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Apr 21 17:42:02.289736 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.289716 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Apr 21 17:42:02.289985 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.289963 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"istio-ca-root-cert\"" Apr 21 17:42:02.290217 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.290200 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Apr 21 17:42:02.290409 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.290225 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z42fm\"" Apr 21 17:42:02.305387 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.305361 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c"] Apr 21 17:42:02.312173 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.312148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.312283 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.312179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.312283 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.312216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.312283 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.312260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttp5\" (UniqueName: \"kubernetes.io/projected/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-kube-api-access-cttp5\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.312485 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.312290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.312485 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.312346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.312485 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.312396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.412901 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.412869 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.412901 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.412902 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.413121 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.412922 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.413121 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.412949 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.413121 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.413098 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.413259 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.413147 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.413259 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.413201 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cttp5\" (UniqueName: \"kubernetes.io/projected/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-kube-api-access-cttp5\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.413609 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.413576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.415155 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.415127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.415441 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.415389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.415641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.415623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.415685 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.415644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.421548 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.421526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.421795 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.421776 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttp5\" (UniqueName: \"kubernetes.io/projected/36c7cb35-cdb1-4065-964f-8f4f97f9d4ba-kube-api-access-cttp5\") pod \"istiod-openshift-gateway-55ff986f96-pgv4c\" (UID: \"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.597786 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.597690 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:02.741112 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:02.741083 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c"] Apr 21 17:42:02.744088 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:42:02.744047 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c7cb35_cdb1_4065_964f_8f4f97f9d4ba.slice/crio-76af5bb9cc180fbe925229bb90d3289962701a92e42a7f9c338bae30544a5d21 WatchSource:0}: Error finding container 76af5bb9cc180fbe925229bb90d3289962701a92e42a7f9c338bae30544a5d21: Status 404 returned error can't find the container with id 76af5bb9cc180fbe925229bb90d3289962701a92e42a7f9c338bae30544a5d21 Apr 21 17:42:03.417235 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:03.417198 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" event={"ID":"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba","Type":"ContainerStarted","Data":"76af5bb9cc180fbe925229bb90d3289962701a92e42a7f9c338bae30544a5d21"} Apr 21 17:42:05.389593 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:05.389555 2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Apr 21 17:42:05.389912 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:05.389622 2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Apr 21 17:42:06.428661 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:06.428625 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" event={"ID":"36c7cb35-cdb1-4065-964f-8f4f97f9d4ba","Type":"ContainerStarted","Data":"2fa83e107cec4d1025bdeb97bb685a2ba9a1b698f6c05437e7d8675045468d7d"} Apr 21 17:42:06.429140 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:06.428910 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:06.430755 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:06.430731 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" Apr 21 17:42:06.450464 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:06.450342 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-pgv4c" podStartSLOduration=1.807654521 podStartE2EDuration="4.450325539s" podCreationTimestamp="2026-04-21 17:42:02 +0000 UTC" firstStartedPulling="2026-04-21 17:42:02.7466299 +0000 UTC m=+526.468744856" lastFinishedPulling="2026-04-21 17:42:05.38930091 +0000 UTC m=+529.111415874" observedRunningTime="2026-04-21 17:42:06.449471007 +0000 UTC m=+530.171585981" watchObservedRunningTime="2026-04-21 17:42:06.450325539 +0000 UTC m=+530.172440514" Apr 21 17:42:10.409086 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:10.409057 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-klhcw" Apr 21 17:42:24.388289 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:42:24.388258 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-856948b99f-kxqxm" Apr 21 17:43:16.737122 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:16.737093 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:43:16.739545 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:16.739524 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:43:18.397018 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.396982 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7"] Apr 21 17:43:18.399663 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.399644 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.403642 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.403618 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Apr 21 17:43:18.403739 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.403641 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jn8hf\"" Apr 21 17:43:18.403958 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.403941 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Apr 21 17:43:18.411699 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.411678 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7"] Apr 21 17:43:18.485083 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.485056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvctk\" (UniqueName: \"kubernetes.io/projected/bb26a9e4-9232-4dce-9053-1c45588c45ac-kube-api-access-rvctk\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.485191 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.485093 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb26a9e4-9232-4dce-9053-1c45588c45ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.586288 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.586262 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb26a9e4-9232-4dce-9053-1c45588c45ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.586395 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.586329 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rvctk\" (UniqueName: \"kubernetes.io/projected/bb26a9e4-9232-4dce-9053-1c45588c45ac-kube-api-access-rvctk\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.586648 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.586629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb26a9e4-9232-4dce-9053-1c45588c45ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.595154 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.595130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvctk\" (UniqueName: \"kubernetes.io/projected/bb26a9e4-9232-4dce-9053-1c45588c45ac-kube-api-access-rvctk\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.710574 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.710509 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:18.840380 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:18.840350 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7"] Apr 21 17:43:18.843647 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:43:18.843610 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb26a9e4_9232_4dce_9053_1c45588c45ac.slice/crio-0756f6be9d207e3636db5d63a5eecab873a75d7ae1366223e4a7ce20fdda63f3 WatchSource:0}: Error finding container 0756f6be9d207e3636db5d63a5eecab873a75d7ae1366223e4a7ce20fdda63f3: Status 404 returned error can't find the container with id 0756f6be9d207e3636db5d63a5eecab873a75d7ae1366223e4a7ce20fdda63f3 Apr 21 17:43:19.652803 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:19.652767 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" event={"ID":"bb26a9e4-9232-4dce-9053-1c45588c45ac","Type":"ContainerStarted","Data":"0756f6be9d207e3636db5d63a5eecab873a75d7ae1366223e4a7ce20fdda63f3"} Apr 21 17:43:24.675322 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:24.675294 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" event={"ID":"bb26a9e4-9232-4dce-9053-1c45588c45ac","Type":"ContainerStarted","Data":"5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64"} Apr 21 17:43:24.675639 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:24.675543 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:24.696819 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:24.696779 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" podStartSLOduration=0.952823668 podStartE2EDuration="6.696768495s" podCreationTimestamp="2026-04-21 17:43:18 +0000 UTC" firstStartedPulling="2026-04-21 17:43:18.845819556 +0000 UTC m=+602.567934508" lastFinishedPulling="2026-04-21 17:43:24.589764371 +0000 UTC m=+608.311879335" observedRunningTime="2026-04-21 17:43:24.695572731 +0000 UTC m=+608.417687704" watchObservedRunningTime="2026-04-21 17:43:24.696768495 +0000 UTC m=+608.418883466" Apr 21 17:43:35.680679 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:35.680597 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:37.201060 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.201024 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8"] Apr 21 17:43:37.203118 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.203103 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.228310 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.228288 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8"] Apr 21 17:43:37.338468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.338432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzpt\" (UniqueName: \"kubernetes.io/projected/f64d6ed3-1261-41c0-94b6-acf44a1540a1-kube-api-access-7qzpt\") pod \"kuadrant-operator-controller-manager-84b657d985-68nx8\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.338634 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.338481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64d6ed3-1261-41c0-94b6-acf44a1540a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-84b657d985-68nx8\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.363540 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.363513 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8"] Apr 21 17:43:37.363752 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:43:37.363727 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-7qzpt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" podUID="f64d6ed3-1261-41c0-94b6-acf44a1540a1" Apr 21 17:43:37.439008 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.438977 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzpt\" (UniqueName: \"kubernetes.io/projected/f64d6ed3-1261-41c0-94b6-acf44a1540a1-kube-api-access-7qzpt\") pod \"kuadrant-operator-controller-manager-84b657d985-68nx8\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.439168 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.439021 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64d6ed3-1261-41c0-94b6-acf44a1540a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-84b657d985-68nx8\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.439269 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.439245 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7"] Apr 21 17:43:37.439387 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.439369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64d6ed3-1261-41c0-94b6-acf44a1540a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-84b657d985-68nx8\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.439501 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.439462 2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" containerName="manager" containerID="cri-o://5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64" gracePeriod=2 Apr 21 17:43:37.453538 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.453478 2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7"] Apr 21 17:43:37.460333 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.460310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzpt\" (UniqueName: \"kubernetes.io/projected/f64d6ed3-1261-41c0-94b6-acf44a1540a1-kube-api-access-7qzpt\") pod \"kuadrant-operator-controller-manager-84b657d985-68nx8\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.473169 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.472714 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz"] Apr 21 17:43:37.473273 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.473182 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" containerName="manager" Apr 21 17:43:37.473273 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.473198 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" containerName="manager" Apr 21 17:43:37.473378 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.473285 2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" containerName="manager" Apr 21 17:43:37.475100 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.475083 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.477733 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.477712 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8"] Apr 21 17:43:37.493566 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.493541 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz"] Apr 21 17:43:37.494840 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.494823 2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8"] Apr 21 17:43:37.522143 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.522117 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.531719 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.531697 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9"] Apr 21 17:43:37.534334 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.534317 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" Apr 21 17:43:37.537493 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.537463 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.538381 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.538353 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vql4l\"" Apr 21 17:43:37.549085 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.549064 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9"] Apr 21 17:43:37.641970 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.641939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9aa3c7-298d-4abf-8145-383dfd29d53c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-gxfnz\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.642098 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.641978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnl4\" (UniqueName: \"kubernetes.io/projected/9f9aa3c7-298d-4abf-8145-383dfd29d53c-kube-api-access-lgnl4\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-gxfnz\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.642154 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.642089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjwg\" (UniqueName: \"kubernetes.io/projected/0e306739-beb9-4c6e-994d-7ff4dd436b5f-kube-api-access-5bjwg\") pod \"limitador-operator-controller-manager-85c4996f8c-sppb9\" (UID: \"0e306739-beb9-4c6e-994d-7ff4dd436b5f\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" Apr 21 17:43:37.683878 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.683857 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:37.686217 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.686190 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.719236 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.719170 2578 generic.go:358] "Generic (PLEG): container finished" podID="bb26a9e4-9232-4dce-9053-1c45588c45ac" containerID="5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64" exitCode=0 Apr 21 17:43:37.719236 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.719218 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" Apr 21 17:43:37.719367 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.719239 2578 scope.go:117] "RemoveContainer" containerID="5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64" Apr 21 17:43:37.719406 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.719369 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.721556 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.721529 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.723455 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.723434 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.723605 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.723589 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:37.725311 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.725284 2578 status_manager.go:895] "Failed to get status for pod" podUID="f64d6ed3-1261-41c0-94b6-acf44a1540a1" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" err="pods \"kuadrant-operator-controller-manager-84b657d985-68nx8\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.727267 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.727246 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.727353 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.727338 2578 scope.go:117] "RemoveContainer" containerID="5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64" Apr 21 17:43:37.727608 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:43:37.727588 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64\": container with ID starting with 5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64 not found: ID does not exist" containerID="5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64" Apr 21 17:43:37.727650 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.727617 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64"} err="failed to get container status \"5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64\": rpc error: code = NotFound desc = could not find container \"5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64\": container with ID starting with 5e00bbccaa0bed569c06553c552b37ca1dadaa5bb0282470d4ad0f5d193f5b64 not found: ID does not exist" Apr 21 17:43:37.729105 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.729084 2578 status_manager.go:895] "Failed to get status for pod" podUID="f64d6ed3-1261-41c0-94b6-acf44a1540a1" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" err="pods \"kuadrant-operator-controller-manager-84b657d985-68nx8\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:37.743479 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.743454 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjwg\" (UniqueName: \"kubernetes.io/projected/0e306739-beb9-4c6e-994d-7ff4dd436b5f-kube-api-access-5bjwg\") pod \"limitador-operator-controller-manager-85c4996f8c-sppb9\" (UID: \"0e306739-beb9-4c6e-994d-7ff4dd436b5f\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" Apr 21 17:43:37.743569 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.743511 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9aa3c7-298d-4abf-8145-383dfd29d53c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-gxfnz\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.743569 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.743545 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnl4\" (UniqueName: \"kubernetes.io/projected/9f9aa3c7-298d-4abf-8145-383dfd29d53c-kube-api-access-lgnl4\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-gxfnz\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.743905 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.743886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9aa3c7-298d-4abf-8145-383dfd29d53c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-gxfnz\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.752174 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.752151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjwg\" (UniqueName: \"kubernetes.io/projected/0e306739-beb9-4c6e-994d-7ff4dd436b5f-kube-api-access-5bjwg\") pod \"limitador-operator-controller-manager-85c4996f8c-sppb9\" (UID: \"0e306739-beb9-4c6e-994d-7ff4dd436b5f\") " pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" Apr 21 17:43:37.752526 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.752508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnl4\" (UniqueName: \"kubernetes.io/projected/9f9aa3c7-298d-4abf-8145-383dfd29d53c-kube-api-access-lgnl4\") pod \"kuadrant-operator-controller-manager-5f895dd7d5-gxfnz\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.820145 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.820121 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:37.844446 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.844398 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64d6ed3-1261-41c0-94b6-acf44a1540a1-extensions-socket-volume\") pod \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " Apr 21 17:43:37.844579 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.844462 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qzpt\" (UniqueName: \"kubernetes.io/projected/f64d6ed3-1261-41c0-94b6-acf44a1540a1-kube-api-access-7qzpt\") pod \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\" (UID: \"f64d6ed3-1261-41c0-94b6-acf44a1540a1\") " Apr 21 17:43:37.844579 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.844520 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb26a9e4-9232-4dce-9053-1c45588c45ac-extensions-socket-volume\") pod \"bb26a9e4-9232-4dce-9053-1c45588c45ac\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " Apr 21 17:43:37.844579 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.844559 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvctk\" (UniqueName: \"kubernetes.io/projected/bb26a9e4-9232-4dce-9053-1c45588c45ac-kube-api-access-rvctk\") pod \"bb26a9e4-9232-4dce-9053-1c45588c45ac\" (UID: \"bb26a9e4-9232-4dce-9053-1c45588c45ac\") " Apr 21 17:43:37.844742 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.844688 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64d6ed3-1261-41c0-94b6-acf44a1540a1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f64d6ed3-1261-41c0-94b6-acf44a1540a1" (UID: "f64d6ed3-1261-41c0-94b6-acf44a1540a1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 21 17:43:37.844829 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.844814 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f64d6ed3-1261-41c0-94b6-acf44a1540a1-extensions-socket-volume\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:43:37.844964 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.844939 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb26a9e4-9232-4dce-9053-1c45588c45ac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bb26a9e4-9232-4dce-9053-1c45588c45ac" (UID: "bb26a9e4-9232-4dce-9053-1c45588c45ac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 21 17:43:37.846543 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.846521 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64d6ed3-1261-41c0-94b6-acf44a1540a1-kube-api-access-7qzpt" (OuterVolumeSpecName: "kube-api-access-7qzpt") pod "f64d6ed3-1261-41c0-94b6-acf44a1540a1" (UID: "f64d6ed3-1261-41c0-94b6-acf44a1540a1"). InnerVolumeSpecName "kube-api-access-7qzpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:43:37.846647 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.846620 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb26a9e4-9232-4dce-9053-1c45588c45ac-kube-api-access-rvctk" (OuterVolumeSpecName: "kube-api-access-rvctk") pod "bb26a9e4-9232-4dce-9053-1c45588c45ac" (UID: "bb26a9e4-9232-4dce-9053-1c45588c45ac"). InnerVolumeSpecName "kube-api-access-rvctk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:43:37.851763 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.851738 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" Apr 21 17:43:37.946047 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.946013 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bb26a9e4-9232-4dce-9053-1c45588c45ac-extensions-socket-volume\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:43:37.946047 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.946042 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvctk\" (UniqueName: \"kubernetes.io/projected/bb26a9e4-9232-4dce-9053-1c45588c45ac-kube-api-access-rvctk\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:43:37.946047 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.946052 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzpt\" (UniqueName: \"kubernetes.io/projected/f64d6ed3-1261-41c0-94b6-acf44a1540a1-kube-api-access-7qzpt\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:43:37.964962 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.964928 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz"] Apr 21 17:43:37.968845 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:43:37.968817 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9aa3c7_298d_4abf_8145_383dfd29d53c.slice/crio-6d8cfa113026c308065b81ba485a842d222b5b9fc64e6e2a3cb311747aafb586 WatchSource:0}: Error finding container 6d8cfa113026c308065b81ba485a842d222b5b9fc64e6e2a3cb311747aafb586: Status 404 returned error can't find the container with id 6d8cfa113026c308065b81ba485a842d222b5b9fc64e6e2a3cb311747aafb586 Apr 21 17:43:37.988483 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:37.988362 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9"] Apr 21 17:43:37.991382 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:43:37.991354 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e306739_beb9_4c6e_994d_7ff4dd436b5f.slice/crio-69f13a3b9303b3924180628677c2b12adbdacf59052dbadc472d9d27e9c94e11 WatchSource:0}: Error finding container 69f13a3b9303b3924180628677c2b12adbdacf59052dbadc472d9d27e9c94e11: Status 404 returned error can't find the container with id 69f13a3b9303b3924180628677c2b12adbdacf59052dbadc472d9d27e9c94e11 Apr 21 17:43:38.030667 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.030637 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:38.033134 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.033113 2578 status_manager.go:895] "Failed to get status for pod" podUID="f64d6ed3-1261-41c0-94b6-acf44a1540a1" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" err="pods \"kuadrant-operator-controller-manager-84b657d985-68nx8\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:38.724500 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.724392 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" event={"ID":"0e306739-beb9-4c6e-994d-7ff4dd436b5f","Type":"ContainerStarted","Data":"69f13a3b9303b3924180628677c2b12adbdacf59052dbadc472d9d27e9c94e11"} Apr 21 17:43:38.726005 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.725979 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" event={"ID":"9f9aa3c7-298d-4abf-8145-383dfd29d53c","Type":"ContainerStarted","Data":"76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505"} Apr 21 17:43:38.726156 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.726011 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" event={"ID":"9f9aa3c7-298d-4abf-8145-383dfd29d53c","Type":"ContainerStarted","Data":"6d8cfa113026c308065b81ba485a842d222b5b9fc64e6e2a3cb311747aafb586"} Apr 21 17:43:38.726156 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.726094 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:38.727137 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.727112 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" Apr 21 17:43:38.749147 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.749105 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" podStartSLOduration=1.7490907839999998 podStartE2EDuration="1.749090784s" podCreationTimestamp="2026-04-21 17:43:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-21 17:43:38.747298189 +0000 UTC m=+622.469413174" watchObservedRunningTime="2026-04-21 17:43:38.749090784 +0000 UTC m=+622.471205758" Apr 21 17:43:38.749914 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.749888 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:38.751919 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.751893 2578 status_manager.go:895] "Failed to get status for pod" podUID="f64d6ed3-1261-41c0-94b6-acf44a1540a1" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" err="pods \"kuadrant-operator-controller-manager-84b657d985-68nx8\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:38.753927 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.753897 2578 status_manager.go:895] "Failed to get status for pod" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-xx8t7" err="pods \"kuadrant-operator-controller-manager-5f895dd7d5-xx8t7\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:38.755972 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.755945 2578 status_manager.go:895] "Failed to get status for pod" podUID="f64d6ed3-1261-41c0-94b6-acf44a1540a1" pod="kuadrant-system/kuadrant-operator-controller-manager-84b657d985-68nx8" err="pods \"kuadrant-operator-controller-manager-84b657d985-68nx8\" is forbidden: User \"system:node:ip-10-0-143-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-230.ec2.internal' and this object" Apr 21 17:43:38.802070 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.802044 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb26a9e4-9232-4dce-9053-1c45588c45ac" path="/var/lib/kubelet/pods/bb26a9e4-9232-4dce-9053-1c45588c45ac/volumes" Apr 21 17:43:38.802533 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:38.802514 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64d6ed3-1261-41c0-94b6-acf44a1540a1" path="/var/lib/kubelet/pods/f64d6ed3-1261-41c0-94b6-acf44a1540a1/volumes" Apr 21 17:43:39.732382 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:39.732347 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" event={"ID":"0e306739-beb9-4c6e-994d-7ff4dd436b5f","Type":"ContainerStarted","Data":"f4d4ad0a9d5aaf9b4cadc5b053e5dd3c269fb35138393877439ff03638a8f54d"} Apr 21 17:43:39.752184 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:39.752132 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" podStartSLOduration=1.563759992 podStartE2EDuration="2.752114961s" podCreationTimestamp="2026-04-21 17:43:37 +0000 UTC" firstStartedPulling="2026-04-21 17:43:37.9935123 +0000 UTC m=+621.715627252" lastFinishedPulling="2026-04-21 17:43:39.181867269 +0000 UTC m=+622.903982221" observedRunningTime="2026-04-21 17:43:39.752097006 +0000 UTC m=+623.474211977" watchObservedRunningTime="2026-04-21 17:43:39.752114961 +0000 UTC m=+623.474229937" Apr 21 17:43:40.735634 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:40.735600 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" Apr 21 17:43:49.734930 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:49.734898 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:51.742029 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:51.741997 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-85c4996f8c-sppb9" Apr 21 17:43:53.530977 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.530943 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz"] Apr 21 17:43:53.533478 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.531255 2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" podUID="9f9aa3c7-298d-4abf-8145-383dfd29d53c" containerName="manager" containerID="cri-o://76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505" gracePeriod=10 Apr 21 17:43:53.755476 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.755454 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:53.778853 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.778823 2578 generic.go:358] "Generic (PLEG): container finished" podID="9f9aa3c7-298d-4abf-8145-383dfd29d53c" containerID="76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505" exitCode=0 Apr 21 17:43:53.778994 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.778911 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" Apr 21 17:43:53.778994 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.778914 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" event={"ID":"9f9aa3c7-298d-4abf-8145-383dfd29d53c","Type":"ContainerDied","Data":"76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505"} Apr 21 17:43:53.778994 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.778987 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz" event={"ID":"9f9aa3c7-298d-4abf-8145-383dfd29d53c","Type":"ContainerDied","Data":"6d8cfa113026c308065b81ba485a842d222b5b9fc64e6e2a3cb311747aafb586"} Apr 21 17:43:53.779157 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.779010 2578 scope.go:117] "RemoveContainer" containerID="76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505" Apr 21 17:43:53.787481 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.787463 2578 scope.go:117] "RemoveContainer" containerID="76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505" Apr 21 17:43:53.787773 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:43:53.787738 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505\": container with ID starting with 76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505 not found: ID does not exist" containerID="76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505" Apr 21 17:43:53.787821 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.787787 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505"} err="failed to get container status \"76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505\": rpc error: code = NotFound desc = could not find container \"76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505\": container with ID starting with 76d1d57f7eeb153346bc85d7a9ffe37400df3fc6855b06a6aca0b1e287877505 not found: ID does not exist" Apr 21 17:43:53.866520 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.866496 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgnl4\" (UniqueName: \"kubernetes.io/projected/9f9aa3c7-298d-4abf-8145-383dfd29d53c-kube-api-access-lgnl4\") pod \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " Apr 21 17:43:53.866620 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.866547 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9aa3c7-298d-4abf-8145-383dfd29d53c-extensions-socket-volume\") pod \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\" (UID: \"9f9aa3c7-298d-4abf-8145-383dfd29d53c\") " Apr 21 17:43:53.866904 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.866881 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f9aa3c7-298d-4abf-8145-383dfd29d53c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9f9aa3c7-298d-4abf-8145-383dfd29d53c" (UID: "9f9aa3c7-298d-4abf-8145-383dfd29d53c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 21 17:43:53.868498 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.868477 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9aa3c7-298d-4abf-8145-383dfd29d53c-kube-api-access-lgnl4" (OuterVolumeSpecName: "kube-api-access-lgnl4") pod "9f9aa3c7-298d-4abf-8145-383dfd29d53c" (UID: "9f9aa3c7-298d-4abf-8145-383dfd29d53c"). InnerVolumeSpecName "kube-api-access-lgnl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:43:53.967489 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.967459 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgnl4\" (UniqueName: \"kubernetes.io/projected/9f9aa3c7-298d-4abf-8145-383dfd29d53c-kube-api-access-lgnl4\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:43:53.967489 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:53.967486 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f9aa3c7-298d-4abf-8145-383dfd29d53c-extensions-socket-volume\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:43:54.115354 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:54.115328 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz"] Apr 21 17:43:54.133147 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:54.133123 2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f895dd7d5-gxfnz"] Apr 21 17:43:54.802551 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:43:54.802522 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f9aa3c7-298d-4abf-8145-383dfd29d53c" path="/var/lib/kubelet/pods/9f9aa3c7-298d-4abf-8145-383dfd29d53c/volumes" Apr 21 17:44:09.799311 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.799277 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz"] Apr 21 17:44:09.799778 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.799683 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f9aa3c7-298d-4abf-8145-383dfd29d53c" containerName="manager" Apr 21 17:44:09.799778 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.799699 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f9aa3c7-298d-4abf-8145-383dfd29d53c" containerName="manager" Apr 21 17:44:09.799849 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.799783 2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="9f9aa3c7-298d-4abf-8145-383dfd29d53c" containerName="manager" Apr 21 17:44:09.802505 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.802483 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.805257 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.805217 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f9bbx\"" Apr 21 17:44:09.815967 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.815944 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz"] Apr 21 17:44:09.875217 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-credential-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875362 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875362 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-workload-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875362 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-envoy\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875362 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-token\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875362 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbwlp\" (UniqueName: \"kubernetes.io/projected/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-kube-api-access-bbwlp\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875629 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-workload-certs\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875629 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.875629 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.875534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-data\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.976444 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976369 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-credential-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.976635 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976479 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.976635 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976519 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-workload-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.976635 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976556 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-envoy\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.976635 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976586 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-token\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.976635 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976610 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bbwlp\" (UniqueName: \"kubernetes.io/projected/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-kube-api-access-bbwlp\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977033 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976670 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-workload-certs\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977150 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.977060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-workload-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977150 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.977071 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977150 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.977004 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-workload-certs\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977150 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.976853 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-credential-socket\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977150 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.977102 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-data\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977353 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.977309 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-data\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.977801 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.977776 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.978862 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.978839 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-envoy\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.979100 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.979082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.984406 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.984377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-istio-token\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:09.984565 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:09.984447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbwlp\" (UniqueName: \"kubernetes.io/projected/abe91a48-bcd3-4ec3-898d-c4e1b52dcd35-kube-api-access-bbwlp\") pod \"maas-default-gateway-openshift-default-58b6f876-jbxbz\" (UID: \"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:10.116494 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:10.116452 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:10.249456 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:10.249429 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz"] Apr 21 17:44:10.252201 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:44:10.252168 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe91a48_bcd3_4ec3_898d_c4e1b52dcd35.slice/crio-8e68da11ad7e9f4dc4e32d6260a5974bc461cbd02bb8ef9e390c6c2b131197c4 WatchSource:0}: Error finding container 8e68da11ad7e9f4dc4e32d6260a5974bc461cbd02bb8ef9e390c6c2b131197c4: Status 404 returned error can't find the container with id 8e68da11ad7e9f4dc4e32d6260a5974bc461cbd02bb8ef9e390c6c2b131197c4 Apr 21 17:44:10.834633 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:10.834592 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" event={"ID":"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35","Type":"ContainerStarted","Data":"8e68da11ad7e9f4dc4e32d6260a5974bc461cbd02bb8ef9e390c6c2b131197c4"} Apr 21 17:44:12.767725 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:12.767679 2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Apr 21 17:44:12.768009 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:12.767755 2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Apr 21 17:44:12.768009 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:12.767786 2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Apr 21 17:44:12.842807 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:12.842779 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" event={"ID":"abe91a48-bcd3-4ec3-898d-c4e1b52dcd35","Type":"ContainerStarted","Data":"1eb5be42d29c5293e7f9cbc1a744408f32376f2a25d91173d126fe15bacbc3dd"} Apr 21 17:44:12.861377 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:12.861317 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" podStartSLOduration=1.347952636 podStartE2EDuration="3.861302182s" podCreationTimestamp="2026-04-21 17:44:09 +0000 UTC" firstStartedPulling="2026-04-21 17:44:10.254051098 +0000 UTC m=+653.976166050" lastFinishedPulling="2026-04-21 17:44:12.767400644 +0000 UTC m=+656.489515596" observedRunningTime="2026-04-21 17:44:12.860158978 +0000 UTC m=+656.582273954" watchObservedRunningTime="2026-04-21 17:44:12.861302182 +0000 UTC m=+656.583417155" Apr 21 17:44:13.116784 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:13.116757 2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:13.121470 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:13.121445 2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:13.846041 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:13.845965 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:13.846824 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:13.846806 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58b6f876-jbxbz" Apr 21 17:44:14.013497 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.013465 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-5gkpv"] Apr 21 17:44:14.016239 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.016217 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.018827 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.018808 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hdtws\"" Apr 21 17:44:14.018919 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.018812 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Apr 21 17:44:14.024579 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.024560 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-5gkpv"] Apr 21 17:44:14.112470 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.112440 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-5gkpv"] Apr 21 17:44:14.112745 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.112725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grtl\" (UniqueName: \"kubernetes.io/projected/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-kube-api-access-2grtl\") pod \"limitador-limitador-7d549b5b-5gkpv\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.112812 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.112761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-config-file\") pod \"limitador-limitador-7d549b5b-5gkpv\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.214112 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.214075 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-config-file\") pod \"limitador-limitador-7d549b5b-5gkpv\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.214284 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.214166 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2grtl\" (UniqueName: \"kubernetes.io/projected/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-kube-api-access-2grtl\") pod \"limitador-limitador-7d549b5b-5gkpv\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.214738 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.214720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-config-file\") pod \"limitador-limitador-7d549b5b-5gkpv\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.223841 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.223820 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grtl\" (UniqueName: \"kubernetes.io/projected/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-kube-api-access-2grtl\") pod \"limitador-limitador-7d549b5b-5gkpv\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.326208 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.326173 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:14.461935 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.461902 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-5gkpv"] Apr 21 17:44:14.465503 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:44:14.465473 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7d99f5_d1c3_486a_a8fd_27c3ca87deb7.slice/crio-ec188ad53bb2fd64d27b121802891d715924bc98809cd2fc60940b29439d2ba4 WatchSource:0}: Error finding container ec188ad53bb2fd64d27b121802891d715924bc98809cd2fc60940b29439d2ba4: Status 404 returned error can't find the container with id ec188ad53bb2fd64d27b121802891d715924bc98809cd2fc60940b29439d2ba4 Apr 21 17:44:14.849945 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:14.849868 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" event={"ID":"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7","Type":"ContainerStarted","Data":"ec188ad53bb2fd64d27b121802891d715924bc98809cd2fc60940b29439d2ba4"} Apr 21 17:44:17.861105 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:17.861064 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" event={"ID":"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7","Type":"ContainerStarted","Data":"af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87"} Apr 21 17:44:17.861573 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:17.861214 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:17.878048 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:17.878001 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" podStartSLOduration=2.167569027 podStartE2EDuration="4.877989239s" podCreationTimestamp="2026-04-21 17:44:13 +0000 UTC" firstStartedPulling="2026-04-21 17:44:14.467402476 +0000 UTC m=+658.189517427" lastFinishedPulling="2026-04-21 17:44:17.177822684 +0000 UTC m=+660.899937639" observedRunningTime="2026-04-21 17:44:17.876316892 +0000 UTC m=+661.598431868" watchObservedRunningTime="2026-04-21 17:44:17.877989239 +0000 UTC m=+661.600104215" Apr 21 17:44:28.637969 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:28.637936 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-5gkpv"] Apr 21 17:44:28.639086 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:28.638223 2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" podUID="5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" containerName="limitador" containerID="cri-o://af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87" gracePeriod=30 Apr 21 17:44:28.639370 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:28.639080 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:29.172047 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.172024 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:29.234728 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.234665 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-config-file\") pod \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " Apr 21 17:44:29.234850 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.234757 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2grtl\" (UniqueName: \"kubernetes.io/projected/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-kube-api-access-2grtl\") pod \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\" (UID: \"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7\") " Apr 21 17:44:29.235027 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.235003 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-config-file" (OuterVolumeSpecName: "config-file") pod "5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" (UID: "5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 21 17:44:29.236771 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.236747 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-kube-api-access-2grtl" (OuterVolumeSpecName: "kube-api-access-2grtl") pod "5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" (UID: "5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7"). InnerVolumeSpecName "kube-api-access-2grtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:44:29.336107 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.336075 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-config-file\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:44:29.336107 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.336102 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2grtl\" (UniqueName: \"kubernetes.io/projected/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7-kube-api-access-2grtl\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:44:29.898621 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.898588 2578 generic.go:358] "Generic (PLEG): container finished" podID="5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" containerID="af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87" exitCode=0 Apr 21 17:44:29.899072 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.898636 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" event={"ID":"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7","Type":"ContainerDied","Data":"af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87"} Apr 21 17:44:29.899072 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.898651 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" Apr 21 17:44:29.899072 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.898667 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7d549b5b-5gkpv" event={"ID":"5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7","Type":"ContainerDied","Data":"ec188ad53bb2fd64d27b121802891d715924bc98809cd2fc60940b29439d2ba4"} Apr 21 17:44:29.899072 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.898684 2578 scope.go:117] "RemoveContainer" containerID="af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87" Apr 21 17:44:29.910603 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.910551 2578 scope.go:117] "RemoveContainer" containerID="af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87" Apr 21 17:44:29.910812 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:44:29.910792 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87\": container with ID starting with af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87 not found: ID does not exist" containerID="af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87" Apr 21 17:44:29.910884 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.910825 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87"} err="failed to get container status \"af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87\": rpc error: code = NotFound desc = could not find container \"af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87\": container with ID starting with af5cef4d9f7adb4b0def3175c34934049666d2f142cad083ffdb2b8910e5df87 not found: ID does not exist" Apr 21 17:44:29.923638 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.923616 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-5gkpv"] Apr 21 17:44:29.928071 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:29.928050 2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-7d549b5b-5gkpv"] Apr 21 17:44:30.101382 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.101348 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-868db5846d-r8wvj"] Apr 21 17:44:30.101752 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.101733 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" containerName="limitador" Apr 21 17:44:30.101839 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.101754 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" containerName="limitador" Apr 21 17:44:30.101839 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.101831 2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" containerName="limitador" Apr 21 17:44:30.104851 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.104828 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.107708 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.107686 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Apr 21 17:44:30.107826 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.107796 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8b9wr\"" Apr 21 17:44:30.114866 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.114845 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-868db5846d-r8wvj"] Apr 21 17:44:30.243149 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.243073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ght\" (UniqueName: \"kubernetes.io/projected/434177da-eb31-41d8-a830-4c5195b5f23e-kube-api-access-g2ght\") pod \"postgres-868db5846d-r8wvj\" (UID: \"434177da-eb31-41d8-a830-4c5195b5f23e\") " pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.243149 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.243128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/434177da-eb31-41d8-a830-4c5195b5f23e-data\") pod \"postgres-868db5846d-r8wvj\" (UID: \"434177da-eb31-41d8-a830-4c5195b5f23e\") " pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.343942 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.343903 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ght\" (UniqueName: \"kubernetes.io/projected/434177da-eb31-41d8-a830-4c5195b5f23e-kube-api-access-g2ght\") pod \"postgres-868db5846d-r8wvj\" (UID: \"434177da-eb31-41d8-a830-4c5195b5f23e\") " pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.344113 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.344033 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/434177da-eb31-41d8-a830-4c5195b5f23e-data\") pod \"postgres-868db5846d-r8wvj\" (UID: \"434177da-eb31-41d8-a830-4c5195b5f23e\") " pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.344385 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.344369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/434177da-eb31-41d8-a830-4c5195b5f23e-data\") pod \"postgres-868db5846d-r8wvj\" (UID: \"434177da-eb31-41d8-a830-4c5195b5f23e\") " pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.352627 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.352601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ght\" (UniqueName: \"kubernetes.io/projected/434177da-eb31-41d8-a830-4c5195b5f23e-kube-api-access-g2ght\") pod \"postgres-868db5846d-r8wvj\" (UID: \"434177da-eb31-41d8-a830-4c5195b5f23e\") " pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.415871 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.415842 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:30.532486 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.532461 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-868db5846d-r8wvj"] Apr 21 17:44:30.534862 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:44:30.534835 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434177da_eb31_41d8_a830_4c5195b5f23e.slice/crio-0dd1b8082dc804de174bfe65bb0c1437591c18917716fec407d3310a10618ed3 WatchSource:0}: Error finding container 0dd1b8082dc804de174bfe65bb0c1437591c18917716fec407d3310a10618ed3: Status 404 returned error can't find the container with id 0dd1b8082dc804de174bfe65bb0c1437591c18917716fec407d3310a10618ed3 Apr 21 17:44:30.801784 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.801708 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7" path="/var/lib/kubelet/pods/5b7d99f5-d1c3-486a-a8fd-27c3ca87deb7/volumes" Apr 21 17:44:30.903217 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:30.903182 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-868db5846d-r8wvj" event={"ID":"434177da-eb31-41d8-a830-4c5195b5f23e","Type":"ContainerStarted","Data":"0dd1b8082dc804de174bfe65bb0c1437591c18917716fec407d3310a10618ed3"} Apr 21 17:44:35.923857 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:35.923820 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-868db5846d-r8wvj" event={"ID":"434177da-eb31-41d8-a830-4c5195b5f23e","Type":"ContainerStarted","Data":"8bf7ffe501639c0df676d4c55bfb9d8a325669da6125867949d5d11643016b76"} Apr 21 17:44:35.924238 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:35.923934 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:35.939832 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:35.939788 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/postgres-868db5846d-r8wvj" podStartSLOduration=0.898861938 podStartE2EDuration="5.939776159s" podCreationTimestamp="2026-04-21 17:44:30 +0000 UTC" firstStartedPulling="2026-04-21 17:44:30.536250892 +0000 UTC m=+674.258365844" lastFinishedPulling="2026-04-21 17:44:35.577165108 +0000 UTC m=+679.299280065" observedRunningTime="2026-04-21 17:44:35.938794285 +0000 UTC m=+679.660909262" watchObservedRunningTime="2026-04-21 17:44:35.939776159 +0000 UTC m=+679.661891132" Apr 21 17:44:41.953854 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:41.953825 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-868db5846d-r8wvj" Apr 21 17:44:45.447023 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.446988 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f5cd555cc-nxvl9"] Apr 21 17:44:45.555029 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.554999 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f5cd555cc-nxvl9"] Apr 21 17:44:45.555184 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.555112 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" Apr 21 17:44:45.557974 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.557948 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lnh65\"" Apr 21 17:44:45.572035 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.572014 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f5cd555cc-nxvl9"] Apr 21 17:44:45.572211 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:44:45.572195 2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gwgjw], unattached volumes=[], failed to process volumes=[kube-api-access-gwgjw]: context canceled" pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" podUID="0839f705-0b22-4c27-a94d-659364924a9e" Apr 21 17:44:45.667612 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.667571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwgjw\" (UniqueName: \"kubernetes.io/projected/0839f705-0b22-4c27-a94d-659364924a9e-kube-api-access-gwgjw\") pod \"maas-controller-5f5cd555cc-nxvl9\" (UID: \"0839f705-0b22-4c27-a94d-659364924a9e\") " pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" Apr 21 17:44:45.768016 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.767941 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gwgjw\" (UniqueName: \"kubernetes.io/projected/0839f705-0b22-4c27-a94d-659364924a9e-kube-api-access-gwgjw\") pod \"maas-controller-5f5cd555cc-nxvl9\" (UID: \"0839f705-0b22-4c27-a94d-659364924a9e\") " pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" Apr 21 17:44:45.776905 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.776882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwgjw\" (UniqueName: \"kubernetes.io/projected/0839f705-0b22-4c27-a94d-659364924a9e-kube-api-access-gwgjw\") pod \"maas-controller-5f5cd555cc-nxvl9\" (UID: \"0839f705-0b22-4c27-a94d-659364924a9e\") " pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" Apr 21 17:44:45.954830 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.954798 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" Apr 21 17:44:45.959606 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:45.959585 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" Apr 21 17:44:46.070689 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:46.070619 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwgjw\" (UniqueName: \"kubernetes.io/projected/0839f705-0b22-4c27-a94d-659364924a9e-kube-api-access-gwgjw\") pod \"0839f705-0b22-4c27-a94d-659364924a9e\" (UID: \"0839f705-0b22-4c27-a94d-659364924a9e\") " Apr 21 17:44:46.072651 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:46.072622 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0839f705-0b22-4c27-a94d-659364924a9e-kube-api-access-gwgjw" (OuterVolumeSpecName: "kube-api-access-gwgjw") pod "0839f705-0b22-4c27-a94d-659364924a9e" (UID: "0839f705-0b22-4c27-a94d-659364924a9e"). InnerVolumeSpecName "kube-api-access-gwgjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:44:46.171191 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:46.171160 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwgjw\" (UniqueName: \"kubernetes.io/projected/0839f705-0b22-4c27-a94d-659364924a9e-kube-api-access-gwgjw\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:44:46.957853 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:46.957819 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f5cd555cc-nxvl9" Apr 21 17:44:46.985539 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:46.985515 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f5cd555cc-nxvl9"] Apr 21 17:44:46.989344 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:46.989320 2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f5cd555cc-nxvl9"] Apr 21 17:44:48.801962 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:48.801931 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0839f705-0b22-4c27-a94d-659364924a9e" path="/var/lib/kubelet/pods/0839f705-0b22-4c27-a94d-659364924a9e/volumes" Apr 21 17:44:51.095574 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.095539 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5f9588dbb6-t47nk"] Apr 21 17:44:51.106342 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.106298 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.108535 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.108512 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5f9588dbb6-t47nk"] Apr 21 17:44:51.108793 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.108775 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Apr 21 17:44:51.108897 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.108778 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-fq944\"" Apr 21 17:44:51.108897 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.108779 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Apr 21 17:44:51.210080 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.210047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f584f9-bc19-4d9d-9d72-725f1171980f-maas-api-tls\") pod \"maas-api-5f9588dbb6-t47nk\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.210233 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.210150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk6vt\" (UniqueName: \"kubernetes.io/projected/e9f584f9-bc19-4d9d-9d72-725f1171980f-kube-api-access-nk6vt\") pod \"maas-api-5f9588dbb6-t47nk\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.311237 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.311207 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nk6vt\" (UniqueName: \"kubernetes.io/projected/e9f584f9-bc19-4d9d-9d72-725f1171980f-kube-api-access-nk6vt\") pod \"maas-api-5f9588dbb6-t47nk\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.311409 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.311250 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f584f9-bc19-4d9d-9d72-725f1171980f-maas-api-tls\") pod \"maas-api-5f9588dbb6-t47nk\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.313651 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.313623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f584f9-bc19-4d9d-9d72-725f1171980f-maas-api-tls\") pod \"maas-api-5f9588dbb6-t47nk\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.319662 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.319629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk6vt\" (UniqueName: \"kubernetes.io/projected/e9f584f9-bc19-4d9d-9d72-725f1171980f-kube-api-access-nk6vt\") pod \"maas-api-5f9588dbb6-t47nk\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.417549 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.417519 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:51.561224 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.561189 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5f9588dbb6-t47nk"] Apr 21 17:44:51.564333 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:44:51.564309 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f584f9_bc19_4d9d_9d72_725f1171980f.slice/crio-3ac98195fd46ded01b3813362a88e4b785c80aeb42644d1cd2985f7adddcde00 WatchSource:0}: Error finding container 3ac98195fd46ded01b3813362a88e4b785c80aeb42644d1cd2985f7adddcde00: Status 404 returned error can't find the container with id 3ac98195fd46ded01b3813362a88e4b785c80aeb42644d1cd2985f7adddcde00 Apr 21 17:44:51.975491 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:51.975449 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5f9588dbb6-t47nk" event={"ID":"e9f584f9-bc19-4d9d-9d72-725f1171980f","Type":"ContainerStarted","Data":"3ac98195fd46ded01b3813362a88e4b785c80aeb42644d1cd2985f7adddcde00"} Apr 21 17:44:54.986698 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:54.986663 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5f9588dbb6-t47nk" event={"ID":"e9f584f9-bc19-4d9d-9d72-725f1171980f","Type":"ContainerStarted","Data":"fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865"} Apr 21 17:44:54.987064 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:54.986789 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:44:55.003588 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:44:55.003541 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-api-5f9588dbb6-t47nk" podStartSLOduration=1.510581995 podStartE2EDuration="4.003527741s" podCreationTimestamp="2026-04-21 17:44:51 +0000 UTC" firstStartedPulling="2026-04-21 17:44:51.56594205 +0000 UTC m=+695.288057002" lastFinishedPulling="2026-04-21 17:44:54.058887791 +0000 UTC m=+697.781002748" observedRunningTime="2026-04-21 17:44:55.001661249 +0000 UTC m=+698.723776223" watchObservedRunningTime="2026-04-21 17:44:55.003527741 +0000 UTC m=+698.725642715" Apr 21 17:45:00.995678 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:00.995611 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:45:22.877792 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.877753 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn"] Apr 21 17:45:22.880977 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.880958 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:22.884895 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.884872 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nnhrd\"" Apr 21 17:45:22.885014 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.884877 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Apr 21 17:45:22.885014 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.884877 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Apr 21 17:45:22.885014 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.884881 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Apr 21 17:45:22.892227 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.892207 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn"] Apr 21 17:45:22.975269 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.975239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpqn\" (UniqueName: \"kubernetes.io/projected/d1b33eda-9651-45e5-a437-36a9de3dacc1-kube-api-access-kbpqn\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:22.975422 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.975288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:22.975422 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.975310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-kserve-provision-location\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:22.975422 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.975393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b33eda-9651-45e5-a437-36a9de3dacc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:22.975539 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.975470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:22.975539 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:22.975494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-home\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.076742 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.076709 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpqn\" (UniqueName: \"kubernetes.io/projected/d1b33eda-9651-45e5-a437-36a9de3dacc1-kube-api-access-kbpqn\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.076923 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.076757 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.076923 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.076775 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-kserve-provision-location\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.076923 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.076818 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b33eda-9651-45e5-a437-36a9de3dacc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.076923 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.076847 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.076923 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.076869 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-home\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.077446 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.077166 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.077446 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.077255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-home\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.077446 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.077273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-kserve-provision-location\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.078954 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.078932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b33eda-9651-45e5-a437-36a9de3dacc1-dshm\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.079310 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.079292 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b33eda-9651-45e5-a437-36a9de3dacc1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.085601 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.085581 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpqn\" (UniqueName: \"kubernetes.io/projected/d1b33eda-9651-45e5-a437-36a9de3dacc1-kube-api-access-kbpqn\") pod \"premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn\" (UID: \"d1b33eda-9651-45e5-a437-36a9de3dacc1\") " pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.190129 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.190048 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:23.321306 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:23.321281 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn"] Apr 21 17:45:23.323430 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:45:23.323391 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b33eda_9651_45e5_a437_36a9de3dacc1.slice/crio-8ced76e2161978c2475af91f37c1298d16c3dd334e597c0c0e1d05181892c0a6 WatchSource:0}: Error finding container 8ced76e2161978c2475af91f37c1298d16c3dd334e597c0c0e1d05181892c0a6: Status 404 returned error can't find the container with id 8ced76e2161978c2475af91f37c1298d16c3dd334e597c0c0e1d05181892c0a6 Apr 21 17:45:24.078900 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:24.078868 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" event={"ID":"d1b33eda-9651-45e5-a437-36a9de3dacc1","Type":"ContainerStarted","Data":"8ced76e2161978c2475af91f37c1298d16c3dd334e597c0c0e1d05181892c0a6"} Apr 21 17:45:29.525908 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:29.525868 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-5f9588dbb6-t47nk"] Apr 21 17:45:29.526348 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:29.526194 2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="opendatahub/maas-api-5f9588dbb6-t47nk" podUID="e9f584f9-bc19-4d9d-9d72-725f1171980f" containerName="maas-api" containerID="cri-o://fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865" gracePeriod=30 Apr 21 17:45:30.916544 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:30.916522 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:45:31.045913 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.045825 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f584f9-bc19-4d9d-9d72-725f1171980f-maas-api-tls\") pod \"e9f584f9-bc19-4d9d-9d72-725f1171980f\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " Apr 21 17:45:31.046078 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.045918 2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk6vt\" (UniqueName: \"kubernetes.io/projected/e9f584f9-bc19-4d9d-9d72-725f1171980f-kube-api-access-nk6vt\") pod \"e9f584f9-bc19-4d9d-9d72-725f1171980f\" (UID: \"e9f584f9-bc19-4d9d-9d72-725f1171980f\") " Apr 21 17:45:31.048022 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.047992 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f584f9-bc19-4d9d-9d72-725f1171980f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9f584f9-bc19-4d9d-9d72-725f1171980f" (UID: "e9f584f9-bc19-4d9d-9d72-725f1171980f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 21 17:45:31.048022 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.048013 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f584f9-bc19-4d9d-9d72-725f1171980f-kube-api-access-nk6vt" (OuterVolumeSpecName: "kube-api-access-nk6vt") pod "e9f584f9-bc19-4d9d-9d72-725f1171980f" (UID: "e9f584f9-bc19-4d9d-9d72-725f1171980f"). InnerVolumeSpecName "kube-api-access-nk6vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 21 17:45:31.105286 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.105248 2578 generic.go:358] "Generic (PLEG): container finished" podID="e9f584f9-bc19-4d9d-9d72-725f1171980f" containerID="fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865" exitCode=0 Apr 21 17:45:31.105465 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.105317 2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5f9588dbb6-t47nk" Apr 21 17:45:31.105465 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.105329 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5f9588dbb6-t47nk" event={"ID":"e9f584f9-bc19-4d9d-9d72-725f1171980f","Type":"ContainerDied","Data":"fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865"} Apr 21 17:45:31.105465 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.105363 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5f9588dbb6-t47nk" event={"ID":"e9f584f9-bc19-4d9d-9d72-725f1171980f","Type":"ContainerDied","Data":"3ac98195fd46ded01b3813362a88e4b785c80aeb42644d1cd2985f7adddcde00"} Apr 21 17:45:31.105465 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.105383 2578 scope.go:117] "RemoveContainer" containerID="fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865" Apr 21 17:45:31.106933 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.106908 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" event={"ID":"d1b33eda-9651-45e5-a437-36a9de3dacc1","Type":"ContainerStarted","Data":"13e21e546b8e68aaeb600bb3e45db5a5a1e75779cd31df587762191a04dea10a"} Apr 21 17:45:31.114439 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.114405 2578 scope.go:117] "RemoveContainer" containerID="fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865" Apr 21 17:45:31.114805 ip-10-0-143-230 kubenswrapper[2578]: E0421 17:45:31.114787 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865\": container with ID starting with fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865 not found: ID does not exist" containerID="fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865" Apr 21 17:45:31.114869 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.114817 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865"} err="failed to get container status \"fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865\": rpc error: code = NotFound desc = could not find container \"fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865\": container with ID starting with fee98b69dd7b56e19cf60f6e30242579944872dd0e468bfd437ae8c3c8664865 not found: ID does not exist" Apr 21 17:45:31.140049 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.140018 2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-5f9588dbb6-t47nk"] Apr 21 17:45:31.144336 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.144312 2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-5f9588dbb6-t47nk"] Apr 21 17:45:31.147124 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.147098 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f584f9-bc19-4d9d-9d72-725f1171980f-maas-api-tls\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:45:31.147124 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:31.147122 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk6vt\" (UniqueName: \"kubernetes.io/projected/e9f584f9-bc19-4d9d-9d72-725f1171980f-kube-api-access-nk6vt\") on node \"ip-10-0-143-230.ec2.internal\" DevicePath \"\"" Apr 21 17:45:32.804205 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:32.804174 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f584f9-bc19-4d9d-9d72-725f1171980f" path="/var/lib/kubelet/pods/e9f584f9-bc19-4d9d-9d72-725f1171980f/volumes" Apr 21 17:45:39.138268 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:39.138233 2578 generic.go:358] "Generic (PLEG): container finished" podID="d1b33eda-9651-45e5-a437-36a9de3dacc1" containerID="13e21e546b8e68aaeb600bb3e45db5a5a1e75779cd31df587762191a04dea10a" exitCode=0 Apr 21 17:45:39.138649 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:39.138281 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" event={"ID":"d1b33eda-9651-45e5-a437-36a9de3dacc1","Type":"ContainerDied","Data":"13e21e546b8e68aaeb600bb3e45db5a5a1e75779cd31df587762191a04dea10a"} Apr 21 17:45:41.146518 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:41.146484 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" event={"ID":"d1b33eda-9651-45e5-a437-36a9de3dacc1","Type":"ContainerStarted","Data":"215acd302c3088e63301e4845cfaabe084ac67e37fd9e06bcc69f73e8a8a884a"} Apr 21 17:45:41.146881 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:41.146694 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:45:41.164980 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:41.164925 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" podStartSLOduration=2.277565588 podStartE2EDuration="19.164908869s" podCreationTimestamp="2026-04-21 17:45:22 +0000 UTC" firstStartedPulling="2026-04-21 17:45:23.325161775 +0000 UTC m=+727.047276727" lastFinishedPulling="2026-04-21 17:45:40.212505054 +0000 UTC m=+743.934620008" observedRunningTime="2026-04-21 17:45:41.163473767 +0000 UTC m=+744.885588733" watchObservedRunningTime="2026-04-21 17:45:41.164908869 +0000 UTC m=+744.887023847" Apr 21 17:45:52.162151 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:45:52.162119 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn" Apr 21 17:46:35.479259 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.479173 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq"] Apr 21 17:46:35.479710 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.479643 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f584f9-bc19-4d9d-9d72-725f1171980f" containerName="maas-api" Apr 21 17:46:35.479710 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.479660 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f584f9-bc19-4d9d-9d72-725f1171980f" containerName="maas-api" Apr 21 17:46:35.479798 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.479731 2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9f584f9-bc19-4d9d-9d72-725f1171980f" containerName="maas-api" Apr 21 17:46:35.483066 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.483048 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.485959 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.485938 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Apr 21 17:46:35.490939 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.490837 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq"] Apr 21 17:46:35.609262 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.609232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-home\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.609447 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.609273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9zj\" (UniqueName: \"kubernetes.io/projected/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-kube-api-access-5w9zj\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.609447 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.609313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.609447 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.609348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-dshm\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.609447 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.609367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-model-cache\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.609447 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.609436 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710085 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710042 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710252 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710101 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-dshm\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710252 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710125 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-model-cache\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710252 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710148 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710252 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710173 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-home\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710252 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710196 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9zj\" (UniqueName: \"kubernetes.io/projected/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-kube-api-access-5w9zj\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710543 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710479 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710607 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-home\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.710661 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.710643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-model-cache\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.712406 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.712383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-dshm\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.712703 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.712686 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.726634 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.726606 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9zj\" (UniqueName: \"kubernetes.io/projected/c4e2a504-9ca1-4b50-8ab5-82cb757e9758-kube-api-access-5w9zj\") pod \"facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq\" (UID: \"c4e2a504-9ca1-4b50-8ab5-82cb757e9758\") " pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.795121 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.795034 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:35.921852 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.921830 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq"] Apr 21 17:46:35.924480 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:46:35.924450 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e2a504_9ca1_4b50_8ab5_82cb757e9758.slice/crio-8d2fb7d3c4c3a24439471785ad5e5296b6429ea6c5690143ed1d5a7f82b33ad7 WatchSource:0}: Error finding container 8d2fb7d3c4c3a24439471785ad5e5296b6429ea6c5690143ed1d5a7f82b33ad7: Status 404 returned error can't find the container with id 8d2fb7d3c4c3a24439471785ad5e5296b6429ea6c5690143ed1d5a7f82b33ad7 Apr 21 17:46:35.926178 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:35.926161 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 21 17:46:36.325707 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:36.325670 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" event={"ID":"c4e2a504-9ca1-4b50-8ab5-82cb757e9758","Type":"ContainerStarted","Data":"b3b9380aeea2df9a926894786f53185b6a6e31275e43111fb63737648032aa69"} Apr 21 17:46:36.325707 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:36.325708 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" event={"ID":"c4e2a504-9ca1-4b50-8ab5-82cb757e9758","Type":"ContainerStarted","Data":"8d2fb7d3c4c3a24439471785ad5e5296b6429ea6c5690143ed1d5a7f82b33ad7"} Apr 21 17:46:42.354117 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:42.354085 2578 generic.go:358] "Generic (PLEG): container finished" podID="c4e2a504-9ca1-4b50-8ab5-82cb757e9758" containerID="b3b9380aeea2df9a926894786f53185b6a6e31275e43111fb63737648032aa69" exitCode=0 Apr 21 17:46:42.354559 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:42.354155 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" event={"ID":"c4e2a504-9ca1-4b50-8ab5-82cb757e9758","Type":"ContainerDied","Data":"b3b9380aeea2df9a926894786f53185b6a6e31275e43111fb63737648032aa69"} Apr 21 17:46:43.358659 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:43.358624 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" event={"ID":"c4e2a504-9ca1-4b50-8ab5-82cb757e9758","Type":"ContainerStarted","Data":"fb512a7426e3f799b1877b741ef9116042fab73dc37491f5ba29cbfc39fdf78b"} Apr 21 17:46:43.359060 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:43.358964 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:46:43.375679 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:43.375629 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" podStartSLOduration=8.178750299 podStartE2EDuration="8.37561379s" podCreationTimestamp="2026-04-21 17:46:35 +0000 UTC" firstStartedPulling="2026-04-21 17:46:42.354766442 +0000 UTC m=+806.076881394" lastFinishedPulling="2026-04-21 17:46:42.551629929 +0000 UTC m=+806.273744885" observedRunningTime="2026-04-21 17:46:43.375189222 +0000 UTC m=+807.097304199" watchObservedRunningTime="2026-04-21 17:46:43.37561379 +0000 UTC m=+807.097728765" Apr 21 17:46:44.792462 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.792406 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb"] Apr 21 17:46:44.795261 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.795244 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.797783 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.797763 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Apr 21 17:46:44.805946 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.805922 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb"] Apr 21 17:46:44.879300 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.879274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.879468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.879304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd546d8-e8a6-43c1-a89a-52e2c183694f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.879468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.879325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4psnv\" (UniqueName: \"kubernetes.io/projected/7bd546d8-e8a6-43c1-a89a-52e2c183694f-kube-api-access-4psnv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.879468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.879350 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.879468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.879437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.879468 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.879459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.979831 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.979799 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.979831 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.979840 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.980075 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.979864 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.980075 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.979883 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd546d8-e8a6-43c1-a89a-52e2c183694f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.980075 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.979904 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4psnv\" (UniqueName: \"kubernetes.io/projected/7bd546d8-e8a6-43c1-a89a-52e2c183694f-kube-api-access-4psnv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.980075 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.979937 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.980275 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.980252 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.980332 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.980289 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.980401 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.980384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.982066 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.982047 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd546d8-e8a6-43c1-a89a-52e2c183694f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.982338 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.982318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd546d8-e8a6-43c1-a89a-52e2c183694f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:44.988546 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:44.988523 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4psnv\" (UniqueName: \"kubernetes.io/projected/7bd546d8-e8a6-43c1-a89a-52e2c183694f-kube-api-access-4psnv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb\" (UID: \"7bd546d8-e8a6-43c1-a89a-52e2c183694f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:45.107857 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:45.107832 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:45.437963 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:45.437939 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb"] Apr 21 17:46:45.441032 ip-10-0-143-230 kubenswrapper[2578]: W0421 17:46:45.441000 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd546d8_e8a6_43c1_a89a_52e2c183694f.slice/crio-db32b2078a0025f32d46f5935cc85f7a6c124fa35c631a2e774564233df92166 WatchSource:0}: Error finding container db32b2078a0025f32d46f5935cc85f7a6c124fa35c631a2e774564233df92166: Status 404 returned error can't find the container with id db32b2078a0025f32d46f5935cc85f7a6c124fa35c631a2e774564233df92166 Apr 21 17:46:46.370391 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:46.370350 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" event={"ID":"7bd546d8-e8a6-43c1-a89a-52e2c183694f","Type":"ContainerStarted","Data":"6d622ab2153a01a32211d265fe4680a2f3932a7e030e01708bc6c1f864386835"} Apr 21 17:46:46.370391 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:46.370388 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" event={"ID":"7bd546d8-e8a6-43c1-a89a-52e2c183694f","Type":"ContainerStarted","Data":"db32b2078a0025f32d46f5935cc85f7a6c124fa35c631a2e774564233df92166"} Apr 21 17:46:51.391008 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:51.390971 2578 generic.go:358] "Generic (PLEG): container finished" podID="7bd546d8-e8a6-43c1-a89a-52e2c183694f" containerID="6d622ab2153a01a32211d265fe4680a2f3932a7e030e01708bc6c1f864386835" exitCode=0 Apr 21 17:46:51.391446 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:51.391044 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" event={"ID":"7bd546d8-e8a6-43c1-a89a-52e2c183694f","Type":"ContainerDied","Data":"6d622ab2153a01a32211d265fe4680a2f3932a7e030e01708bc6c1f864386835"} Apr 21 17:46:52.396162 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:52.396125 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" event={"ID":"7bd546d8-e8a6-43c1-a89a-52e2c183694f","Type":"ContainerStarted","Data":"8a1d519cb75e27966aa120394c66c8cd552e7c67506dc304c6eaef75e7e886d6"} Apr 21 17:46:52.396605 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:52.396349 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:46:52.414081 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:52.414035 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" podStartSLOduration=8.230858041 podStartE2EDuration="8.414023451s" podCreationTimestamp="2026-04-21 17:46:44 +0000 UTC" firstStartedPulling="2026-04-21 17:46:51.391661438 +0000 UTC m=+815.113776390" lastFinishedPulling="2026-04-21 17:46:51.574826845 +0000 UTC m=+815.296941800" observedRunningTime="2026-04-21 17:46:52.413449217 +0000 UTC m=+816.135564192" watchObservedRunningTime="2026-04-21 17:46:52.414023451 +0000 UTC m=+816.136138469" Apr 21 17:46:54.375108 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:46:54.375072 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq" Apr 21 17:47:03.412833 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:47:03.412804 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb" Apr 21 17:48:16.764152 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:48:16.763359 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:48:16.767638 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:48:16.767612 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:53:16.787558 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:53:16.787527 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:53:16.790643 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:53:16.790621 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:58:16.809641 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:58:16.809529 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 17:58:16.812513 ip-10-0-143-230 kubenswrapper[2578]: I0421 17:58:16.812426 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 18:03:16.831040 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:03:16.830925 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 18:03:16.834482 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:03:16.834449 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 18:08:16.851874 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:08:16.851766 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 18:08:16.857132 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:08:16.857111 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 18:09:32.809484 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:32.809452 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-856948b99f-kxqxm_65669328-4726-4eba-b1d2-88e5025ad1dd/manager/0.log" Apr 21 18:09:33.139688 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:33.139657 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-858dbf95b8-2ptgp_fb301ae7-390b-4a44-93fb-a6b34d17a309/manager/1.log" Apr 21 18:09:33.479308 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:33.479227 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz_ba106c97-f27e-40e0-aa44-9c2a95887924/manager/0.log" Apr 21 18:09:33.584500 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:33.584468 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-868db5846d-r8wvj_434177da-eb31-41d8-a830-4c5195b5f23e/postgres/0.log" Apr 21 18:09:35.647796 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:35.647766 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-85c4996f8c-sppb9_0e306739-beb9-4c6e-994d-7ff4dd436b5f/manager/0.log" Apr 21 18:09:36.095041 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:36.094968 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-pgv4c_36c7cb35-cdb1-4065-964f-8f4f97f9d4ba/discovery/0.log" Apr 21 18:09:36.212734 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:36.212704 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-548b8d8fcb-z2ztv_95b197dd-ee42-4148-9125-dcf0bd52abf1/kube-auth-proxy/0.log" Apr 21 18:09:36.434807 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:36.434774 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58b6f876-jbxbz_abe91a48-bcd3-4ec3-898d-c4e1b52dcd35/istio-proxy/0.log" Apr 21 18:09:37.219355 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:37.219322 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb_7bd546d8-e8a6-43c1-a89a-52e2c183694f/storage-initializer/0.log" Apr 21 18:09:37.226408 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:37.226380 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-75cdccg6bwb_7bd546d8-e8a6-43c1-a89a-52e2c183694f/main/0.log" Apr 21 18:09:37.333544 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:37.333517 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq_c4e2a504-9ca1-4b50-8ab5-82cb757e9758/storage-initializer/0.log" Apr 21 18:09:37.341519 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:37.341493 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-8f8dc67b7-vpbjq_c4e2a504-9ca1-4b50-8ab5-82cb757e9758/main/0.log" Apr 21 18:09:37.444612 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:37.444581 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn_d1b33eda-9651-45e5-a437-36a9de3dacc1/main/0.log" Apr 21 18:09:37.450793 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:37.450768 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-6b97b89985-fdhtn_d1b33eda-9651-45e5-a437-36a9de3dacc1/storage-initializer/0.log" Apr 21 18:09:43.964603 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:43.964571 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8dcr9_dcf8f14a-09d1-4071-94df-ba4064d021a4/global-pull-secret-syncer/0.log" Apr 21 18:09:44.165049 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:44.165020 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vfwzr_b36fb126-3aba-45e0-8d63-ad3846dcc93a/konnectivity-agent/0.log" Apr 21 18:09:44.224990 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:44.224915 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-230.ec2.internal_5e474269ec2989f5a8531df60ebc0fd0/haproxy/0.log" Apr 21 18:09:48.444514 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:48.444480 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-85c4996f8c-sppb9_0e306739-beb9-4c6e-994d-7ff4dd436b5f/manager/0.log" Apr 21 18:09:50.232773 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:50.232743 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trhjt_8e7411a8-d5bb-4c7d-9d23-7785d9751b6a/node-exporter/0.log" Apr 21 18:09:50.252101 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:50.252057 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trhjt_8e7411a8-d5bb-4c7d-9d23-7785d9751b6a/kube-rbac-proxy/0.log" Apr 21 18:09:50.273042 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:50.272986 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-trhjt_8e7411a8-d5bb-4c7d-9d23-7785d9751b6a/init-textfile/0.log" Apr 21 18:09:52.138690 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:52.138653 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-cb95c66f6-kzkr7_dece6707-af2b-4518-b0f4-95ec5f993643/networking-console-plugin/0.log" Apr 21 18:09:53.464642 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.464610 2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst"] Apr 21 18:09:53.467748 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.467730 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.470260 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.470239 2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nf5lv\"/\"default-dockercfg-28xmb\"" Apr 21 18:09:53.470387 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.470300 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nf5lv\"/\"kube-root-ca.crt\"" Apr 21 18:09:53.470387 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.470334 2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nf5lv\"/\"openshift-service-ca.crt\"" Apr 21 18:09:53.478838 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.478819 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst"] Apr 21 18:09:53.554067 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.554040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2g27\" (UniqueName: \"kubernetes.io/projected/23fe22bf-b2c9-4579-bf33-e2da78ac072e-kube-api-access-r2g27\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.554191 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.554080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-sys\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.554191 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.554105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-podres\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.554191 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.554170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-lib-modules\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.554299 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.554206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-proc\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655398 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655369 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-lib-modules\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655520 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655403 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-proc\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655520 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655460 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r2g27\" (UniqueName: \"kubernetes.io/projected/23fe22bf-b2c9-4579-bf33-e2da78ac072e-kube-api-access-r2g27\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655520 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655492 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-sys\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655625 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655521 2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-podres\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655625 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-lib-modules\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655625 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-proc\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655625 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-sys\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.655743 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.655639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23fe22bf-b2c9-4579-bf33-e2da78ac072e-podres\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.663852 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.663833 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2g27\" (UniqueName: \"kubernetes.io/projected/23fe22bf-b2c9-4579-bf33-e2da78ac072e-kube-api-access-r2g27\") pod \"perf-node-gather-daemonset-k8vst\" (UID: \"23fe22bf-b2c9-4579-bf33-e2da78ac072e\") " pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.777546 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.777487 2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:53.893964 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.893939 2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst"] Apr 21 18:09:53.896450 ip-10-0-143-230 kubenswrapper[2578]: W0421 18:09:53.896425 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23fe22bf_b2c9_4579_bf33_e2da78ac072e.slice/crio-0934cbd52efc19d82033fe228a0c66f0f924f94ce14cc62015f99e10640f8e11 WatchSource:0}: Error finding container 0934cbd52efc19d82033fe228a0c66f0f924f94ce14cc62015f99e10640f8e11: Status 404 returned error can't find the container with id 0934cbd52efc19d82033fe228a0c66f0f924f94ce14cc62015f99e10640f8e11 Apr 21 18:09:53.897966 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.897948 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 21 18:09:53.952315 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:53.952289 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" event={"ID":"23fe22bf-b2c9-4579-bf33-e2da78ac072e","Type":"ContainerStarted","Data":"0934cbd52efc19d82033fe228a0c66f0f924f94ce14cc62015f99e10640f8e11"} Apr 21 18:09:54.533832 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:54.533802 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qz45_137fa6ca-d790-417d-bc67-5a19c38af051/dns/0.log" Apr 21 18:09:54.551946 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:54.551929 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4qz45_137fa6ca-d790-417d-bc67-5a19c38af051/kube-rbac-proxy/0.log" Apr 21 18:09:54.663128 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:54.663104 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n859_951bbb2d-337a-4723-9e9c-ef08f471f1fa/dns-node-resolver/0.log" Apr 21 18:09:54.956295 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:54.956262 2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" event={"ID":"23fe22bf-b2c9-4579-bf33-e2da78ac072e","Type":"ContainerStarted","Data":"83cc129a1af398d0558aca9eb864df70ba021c991bf85776a2b325312908d639"} Apr 21 18:09:54.956295 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:54.956305 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:09:54.972606 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:54.972561 2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" podStartSLOduration=1.972546602 podStartE2EDuration="1.972546602s" podCreationTimestamp="2026-04-21 18:09:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-21 18:09:54.970701849 +0000 UTC m=+2198.692816836" watchObservedRunningTime="2026-04-21 18:09:54.972546602 +0000 UTC m=+2198.694661576" Apr 21 18:09:55.281737 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:55.281652 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vcxwk_547c1477-65a3-4469-8fb5-5700ad4cd216/node-ca/0.log" Apr 21 18:09:56.217962 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:56.217928 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-pgv4c_36c7cb35-cdb1-4065-964f-8f4f97f9d4ba/discovery/0.log" Apr 21 18:09:56.234072 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:56.234044 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-548b8d8fcb-z2ztv_95b197dd-ee42-4148-9125-dcf0bd52abf1/kube-auth-proxy/0.log" Apr 21 18:09:56.333956 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:56.333933 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58b6f876-jbxbz_abe91a48-bcd3-4ec3-898d-c4e1b52dcd35/istio-proxy/0.log" Apr 21 18:09:56.896766 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:56.896735 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8bv2c_1b8ca466-7111-427f-891f-efc7ebe9d92d/serve-healthcheck-canary/0.log" Apr 21 18:09:57.584667 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:57.584636 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tz8cg_033302de-ab2b-4b50-8bf5-bea5f1870198/kube-rbac-proxy/0.log" Apr 21 18:09:57.600972 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:57.600944 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tz8cg_033302de-ab2b-4b50-8bf5-bea5f1870198/exporter/0.log" Apr 21 18:09:57.619654 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:57.619635 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tz8cg_033302de-ab2b-4b50-8bf5-bea5f1870198/extractor/0.log" Apr 21 18:09:59.553577 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:59.553542 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-856948b99f-kxqxm_65669328-4726-4eba-b1d2-88e5025ad1dd/manager/0.log" Apr 21 18:09:59.668284 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:59.668251 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-858dbf95b8-2ptgp_fb301ae7-390b-4a44-93fb-a6b34d17a309/manager/0.log" Apr 21 18:09:59.691201 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:59.691170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-858dbf95b8-2ptgp_fb301ae7-390b-4a44-93fb-a6b34d17a309/manager/1.log" Apr 21 18:09:59.783430 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:59.783389 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d5f5c78f5-dz7cz_ba106c97-f27e-40e0-aa44-9c2a95887924/manager/0.log" Apr 21 18:09:59.799793 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:09:59.799772 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-868db5846d-r8wvj_434177da-eb31-41d8-a830-4c5195b5f23e/postgres/0.log" Apr 21 18:10:00.947340 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:00.947307 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-qtxrb_8add952f-3146-4786-a24a-2e0e8f70e91c/openshift-lws-operator/0.log" Apr 21 18:10:00.969454 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:00.969408 2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nf5lv/perf-node-gather-daemonset-k8vst" Apr 21 18:10:06.778978 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:06.778942 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2z52h_b3521509-adc6-48b2-905b-f7597ce17704/kube-multus/0.log" Apr 21 18:10:06.937546 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:06.937514 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9fvp6_c70a5efa-9e1c-4b04-b2df-fa59050fcd7e/kube-multus-additional-cni-plugins/0.log" Apr 21 18:10:06.954952 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:06.954926 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9fvp6_c70a5efa-9e1c-4b04-b2df-fa59050fcd7e/egress-router-binary-copy/0.log" Apr 21 18:10:06.970885 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:06.970859 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9fvp6_c70a5efa-9e1c-4b04-b2df-fa59050fcd7e/cni-plugins/0.log" Apr 21 18:10:06.988925 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:06.988906 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9fvp6_c70a5efa-9e1c-4b04-b2df-fa59050fcd7e/bond-cni-plugin/0.log" Apr 21 18:10:07.009634 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:07.009612 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9fvp6_c70a5efa-9e1c-4b04-b2df-fa59050fcd7e/routeoverride-cni/0.log" Apr 21 18:10:07.030462 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:07.030383 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9fvp6_c70a5efa-9e1c-4b04-b2df-fa59050fcd7e/whereabouts-cni-bincopy/0.log" Apr 21 18:10:07.049942 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:07.049913 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9fvp6_c70a5efa-9e1c-4b04-b2df-fa59050fcd7e/whereabouts-cni/0.log" Apr 21 18:10:07.332001 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:07.331900 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z46hp_cbb61ff1-0bdf-4220-99ee-6f860546dc41/network-metrics-daemon/0.log" Apr 21 18:10:07.346046 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:07.346016 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z46hp_cbb61ff1-0bdf-4220-99ee-6f860546dc41/kube-rbac-proxy/0.log" Apr 21 18:10:08.403959 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.403928 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-controller/0.log" Apr 21 18:10:08.417664 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.417644 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/0.log" Apr 21 18:10:08.438646 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.438618 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovn-acl-logging/1.log" Apr 21 18:10:08.458918 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.458889 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/kube-rbac-proxy-node/0.log" Apr 21 18:10:08.478994 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.478969 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/kube-rbac-proxy-ovn-metrics/0.log" Apr 21 18:10:08.493126 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.493108 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/northd/0.log" Apr 21 18:10:08.508562 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.508528 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/nbdb/0.log" Apr 21 18:10:08.527555 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.527533 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/sbdb/0.log" Apr 21 18:10:08.681896 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:08.681826 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms5gf_abdbee8d-8443-4ccd-a3d5-ef918b3fc39a/ovnkube-controller/0.log" Apr 21 18:10:10.142886 ip-10-0-143-230 kubenswrapper[2578]: I0421 18:10:10.142854 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5d95f_9959f536-1d43-45dc-a1b9-a84e76dbafa0/network-check-target-container/0.log"