Sep 09 23:15:03.051862 ip-10-0-142-167 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 23:15:03.051874 ip-10-0-142-167 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 23:15:03.051884 ip-10-0-142-167 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 23:15:03.052085 ip-10-0-142-167 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 23:15:14.301015 ip-10-0-142-167 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 23:15:14.301034 ip-10-0-142-167 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e523760a0e10475387d5ed884a6a09a7 -- Sep 09 23:16:46.069878 ip-10-0-142-167 systemd[1]: Starting Kubernetes Kubelet... Sep 09 23:16:46.461826 ip-10-0-142-167 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:16:46.461826 ip-10-0-142-167 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 23:16:46.461826 ip-10-0-142-167 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:16:46.461826 ip-10-0-142-167 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 23:16:46.461826 ip-10-0-142-167 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:16:46.462410 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.461870 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 23:16:46.466936 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466919 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:16:46.466936 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466937 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466940 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466943 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466946 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466950 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466952 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466956 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466960 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466964 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466967 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466970 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466973 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466976 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466979 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466982 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466985 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466987 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466990 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466993 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466995 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:16:46.467016 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.466999 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467001 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467004 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467006 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467009 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467012 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467014 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467016 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467019 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467022 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467025 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467028 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467031 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467034 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467037 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467039 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467042 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467044 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467047 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467049 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:16:46.467490 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467052 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467054 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467057 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467059 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467061 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467065 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467067 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467070 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467072 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467075 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467078 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467080 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467083 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467086 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467090 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467093 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467095 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467099 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467101 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:16:46.468108 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467104 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467106 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467109 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467111 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467114 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467116 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467120 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467124 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467126 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467131 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467134 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467137 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467140 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467143 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467146 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467150 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467153 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467155 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467158 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:16:46.468777 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467161 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467164 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467166 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467169 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467171 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467174 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467177 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467179 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467183 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467186 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467189 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467191 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467194 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467197 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467199 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467202 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467204 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467207 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467209 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:16:46.469249 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.467212 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:16:46.469732 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469717 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:16:46.469761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469733 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:16:46.469761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469738 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:16:46.469761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469743 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:16:46.469761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469747 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:16:46.469761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469751 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:16:46.469761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469755 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:16:46.469761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469761 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469766 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469770 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469774 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469777 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469780 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469784 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469788 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469791 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469794 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469797 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469800 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469806 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469809 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469812 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469815 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469817 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469820 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469823 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:16:46.469935 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469825 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469828 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469831 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469834 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469836 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469839 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469841 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469844 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469847 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469849 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469852 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469855 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469858 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469861 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469864 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469867 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469870 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469872 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469875 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469878 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:16:46.470400 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469880 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469883 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469885 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469888 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469892 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469895 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469897 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469900 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469903 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469906 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469908 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469911 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469914 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469916 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469925 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469928 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469931 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469934 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469937 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469940 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:16:46.470908 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469943 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469947 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469949 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469952 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469954 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469958 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469960 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469964 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469967 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469969 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469972 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469975 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469977 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469980 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469982 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469985 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469988 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469991 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469995 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:16:46.471387 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.469999 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470002 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470004 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470007 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470010 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470012 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470015 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470017 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470020 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470022 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470025 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470028 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470030 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470033 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470387 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470392 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470395 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470398 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470401 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470404 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:16:46.471893 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470408 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470410 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470414 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470416 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470419 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470422 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470425 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470428 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470431 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470433 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470437 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470440 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470442 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470446 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470448 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470451 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470453 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470456 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470459 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470462 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:16:46.472377 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470464 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470467 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470470 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470473 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470475 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470478 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470480 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470483 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470486 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470488 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470491 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470493 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470496 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470499 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470501 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470504 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470506 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470509 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470511 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470514 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:16:46.472887 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470516 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470519 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470521 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470524 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470527 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470529 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470533 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470536 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470538 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470540 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470543 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470546 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470549 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470551 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470554 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470557 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470561 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470565 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470568 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:16:46.473398 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470571 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470574 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470576 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470579 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470582 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470585 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470605 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470609 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470612 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470615 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470618 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470620 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470623 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470628 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470631 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470634 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470637 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470642 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470645 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:16:46.473888 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470648 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470651 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470655 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470658 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470661 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470664 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470667 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470670 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470673 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470676 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470679 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470681 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470684 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470687 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.470690 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470767 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470810 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470816 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470821 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470826 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470829 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 23:16:46.474348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470833 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470837 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470841 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470844 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470847 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470850 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470854 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470857 2559 flags.go:64] FLAG: --cgroup-root="" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470860 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470863 2559 flags.go:64] FLAG: --client-ca-file="" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470866 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470869 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470874 2559 flags.go:64] FLAG: --cluster-domain="" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470876 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470885 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470888 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470891 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470898 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470901 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470904 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470907 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470910 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470914 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470916 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 23:16:46.474914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470920 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470922 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470926 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470929 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470932 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470936 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470939 2559 flags.go:64] FLAG: --enable-server="true" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470941 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470945 2559 flags.go:64] FLAG: --event-burst="100" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470948 2559 flags.go:64] FLAG: --event-qps="50" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470951 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470954 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470957 2559 flags.go:64] FLAG: --eviction-hard="" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470960 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470963 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470966 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470969 2559 flags.go:64] FLAG: --eviction-soft="" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470972 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470974 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470978 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470980 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470983 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470986 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470989 2559 flags.go:64] FLAG: --feature-gates="" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470992 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 23:16:46.475478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.470996 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471000 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471003 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471006 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471009 2559 flags.go:64] FLAG: --help="false" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471011 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471015 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471018 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471020 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471024 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471027 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471030 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471033 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471035 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471038 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471041 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471044 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471047 2559 flags.go:64] FLAG: --kube-reserved="" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471049 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471052 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471055 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471058 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471061 2559 flags.go:64] FLAG: --lock-file="" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471064 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 23:16:46.476135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471067 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471070 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471075 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471079 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471082 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471084 2559 flags.go:64] FLAG: --logging-format="text" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471087 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471090 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471094 2559 flags.go:64] FLAG: --manifest-url="" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471097 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471101 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471104 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471108 2559 flags.go:64] FLAG: --max-pods="110" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471111 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471114 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471117 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471120 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471123 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471126 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471129 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471136 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471139 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471142 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 23:16:46.476749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471145 2559 flags.go:64] FLAG: --pod-cidr="" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471148 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471154 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471156 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471159 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471162 2559 flags.go:64] FLAG: --port="10250" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471165 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471168 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e240da9e7929e274" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471171 2559 flags.go:64] FLAG: --qos-reserved="" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471174 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471177 2559 flags.go:64] FLAG: --register-node="true" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471179 2559 flags.go:64] FLAG: --register-with-taints="" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471183 2559 flags.go:64] FLAG: --registry-burst="10" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471186 2559 flags.go:64] FLAG: --registry-qps="5" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471189 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471192 2559 flags.go:64] FLAG: --reserved-memory="" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471196 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471199 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471203 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471206 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471208 2559 flags.go:64] FLAG: --runonce="false" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471211 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471214 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471217 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471220 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 23:16:46.477313 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471223 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471226 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471229 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471232 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471236 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471239 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471242 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471245 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471248 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471251 2559 flags.go:64] FLAG: --system-cgroups="" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471253 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471259 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471262 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471264 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471269 2559 flags.go:64] FLAG: --tls-min-version="" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471271 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471274 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471277 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471280 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471282 2559 flags.go:64] FLAG: --v="2" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471286 2559 flags.go:64] FLAG: --version="false" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471290 2559 flags.go:64] FLAG: --vmodule="" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471294 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.471297 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471406 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:16:46.478112 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471411 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471414 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471417 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471420 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471424 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471428 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471430 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471433 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471436 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471438 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471441 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471445 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471447 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471450 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471453 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471455 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471458 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471460 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471463 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471466 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:16:46.479344 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471468 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471471 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471473 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471476 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471478 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471481 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471484 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471486 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471490 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471494 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471498 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471500 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471503 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471506 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471509 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471511 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471525 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471527 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471530 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:16:46.480113 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471532 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471535 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471537 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471540 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471544 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471547 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471549 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471552 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471555 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471557 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471560 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471563 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471566 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471568 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471571 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471574 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471576 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471579 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471582 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471584 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:16:46.480576 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471603 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471608 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471611 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471615 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471618 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471620 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471625 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471628 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471632 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471634 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471637 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471640 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471642 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471645 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471648 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471650 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471655 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471657 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471660 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:16:46.481285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471662 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471665 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471667 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471670 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471672 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471675 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471678 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471680 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471683 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471685 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471688 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471690 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471693 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471695 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471698 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471701 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471703 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471706 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471709 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:16:46.482124 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.471712 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.472286 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.479882 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.479902 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.480845 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.480865 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480950 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480959 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480964 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480970 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480975 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480980 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480985 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480990 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480995 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:16:46.482761 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.480999 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481003 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481008 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481012 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481016 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481020 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481025 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481029 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481033 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481037 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481042 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481046 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481050 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481055 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481059 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481064 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481068 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481073 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481077 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481081 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:16:46.483407 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481087 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481092 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481096 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481100 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481104 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481108 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481112 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481117 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481124 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481129 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481135 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481139 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481144 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481149 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481154 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481158 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481162 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481166 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481170 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:16:46.484285 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481175 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481179 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481183 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481188 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481192 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481196 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481200 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481204 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481209 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481214 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481219 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481223 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481228 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481232 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481236 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481240 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481245 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481249 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481253 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481259 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:16:46.484863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481264 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481269 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481274 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481278 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481282 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481286 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481290 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481294 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481298 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481301 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481305 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481309 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481314 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481318 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481322 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481326 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481330 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481334 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481338 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481343 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:16:46.485396 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481349 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481354 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481361 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481366 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481370 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481375 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481379 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481384 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481388 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481392 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481397 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.481405 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481579 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481604 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481610 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:16:46.485945 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481615 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481619 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481624 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481628 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481632 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481637 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481641 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481645 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481649 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481653 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481657 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481661 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481665 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481669 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481674 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481678 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481682 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481686 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481690 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481694 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:16:46.486314 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481700 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481704 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481708 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481713 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481717 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481724 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481731 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481736 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481740 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481745 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481749 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481753 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481757 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481761 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481767 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481772 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481776 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481780 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481784 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:16:46.486997 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481788 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481792 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481796 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481801 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481805 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481812 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481818 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481823 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481827 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481832 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481837 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481842 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481847 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481851 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481857 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481861 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481866 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481870 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481874 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481879 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:16:46.487504 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481883 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481887 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481891 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481895 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481899 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481903 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481907 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481911 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481915 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481919 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481923 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481927 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481932 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481936 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481940 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481944 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481948 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481952 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481956 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481960 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:16:46.488141 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481964 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481968 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481973 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481977 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481981 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481985 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481989 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.481998 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482002 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482006 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482010 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482014 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482019 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482023 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482028 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482032 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:16:46.488954 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:46.482037 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:16:46.489346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.482045 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:16:46.489346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.482185 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 23:16:46.489346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.484757 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 23:16:46.489346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.485742 2559 server.go:1019] "Starting client certificate rotation" Sep 09 23:16:46.489346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.485844 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:16:46.489346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.485875 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:16:46.489346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.488482 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:16:46.491578 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.491558 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:16:46.502030 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.502011 2559 log.go:25] "Validated CRI v1 runtime API" Sep 09 23:16:46.505921 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.505904 2559 log.go:25] "Validated CRI v1 image API" Sep 09 23:16:46.506812 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.506795 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 23:16:46.509856 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.509840 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d4bb4e1f-a646-4cb7-93d9-7debf5fbf092:/dev/nvme0n1p3 fd4d4fb4-8c90-4d8b-a839-ae20bc0e1117:/dev/nvme0n1p4] Sep 09 23:16:46.509905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.509856 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 23:16:46.514696 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.514680 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:16:46.514856 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.514753 2559 manager.go:216] Machine: {Timestamp:2026-09-09 23:16:46.513411901 +0000 UTC m=+0.326020253 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100481 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d8806855d0407c8c94ecb8fb71b0f SystemUUID:ec2d8806-855d-0407-c8c9-4ecb8fb71b0f BootID:e523760a-0e10-4753-87d5-ed884a6a09a7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:b8:b7:6c:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:b8:b7:6c:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:01:cf:52:c6:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 23:16:46.514856 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.514853 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 23:16:46.514945 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.514927 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 23:16:46.517847 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.517825 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 23:16:46.517972 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.517849 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-167.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 23:16:46.518519 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.518510 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 23:16:46.518558 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.518522 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 23:16:46.518558 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.518530 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:16:46.519218 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.519208 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:16:46.520387 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.520377 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 23:16:46.523890 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.523879 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:16:46.524002 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.523993 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 23:16:46.524074 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.524023 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 09 23:16:46.524074 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.524036 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 23:16:46.524074 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.524054 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 23:16:46.524074 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.524068 2559 kubelet.go:398] "Adding apiserver pod source" Sep 09 23:16:46.524197 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.524081 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 23:16:46.524458 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.524446 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:16:46.524501 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.524464 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:16:46.526756 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.526738 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 23:16:46.528928 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.528914 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 23:16:46.528979 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.528944 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 23:16:46.529093 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529084 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 23:16:46.529127 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529098 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 23:16:46.529127 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529103 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 23:16:46.529127 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529109 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 23:16:46.529127 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529116 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 23:16:46.529127 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529121 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 23:16:46.529724 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529712 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 23:16:46.529724 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529722 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 23:16:46.529823 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529729 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 23:16:46.529823 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529734 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 23:16:46.529823 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529753 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 23:16:46.529823 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.529761 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 23:16:46.530613 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.530603 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 23:16:46.530613 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.530612 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 23:16:46.534485 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.534470 2559 server.go:1268] "Started kubelet" Sep 09 23:16:46.534602 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.534544 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 23:16:46.534664 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.534623 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 23:16:46.534710 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.534661 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 23:16:46.535062 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.535046 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-167.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 23:16:46.535157 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.535132 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 23:16:46.535281 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.535260 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 23:16:46.535342 ip-10-0-142-167 systemd[1]: Started Kubernetes Kubelet. Sep 09 23:16:46.536531 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.536158 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 23:16:46.538150 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.537972 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 09 23:16:46.539671 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.539642 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 23:16:46.540417 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.540395 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 23:16:46.540503 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.540396 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 23:16:46.540553 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.540512 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:46.541095 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541077 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 23:16:46.541095 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541078 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 23:16:46.541212 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541103 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 23:16:46.541212 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541188 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 09 23:16:46.541212 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541195 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 09 23:16:46.541348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541222 2559 factory.go:55] Registering systemd factory Sep 09 23:16:46.541348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541239 2559 factory.go:223] Registration of the systemd container factory successfully Sep 09 23:16:46.541505 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541491 2559 factory.go:153] Registering CRI-O factory Sep 09 23:16:46.541564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541508 2559 factory.go:223] Registration of the crio container factory successfully Sep 09 23:16:46.541623 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541602 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 23:16:46.541675 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541623 2559 factory.go:103] Registering Raw factory Sep 09 23:16:46.541675 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.541638 2559 manager.go:1196] Started watching for new ooms in manager Sep 09 23:16:46.542047 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.542034 2559 manager.go:349] Starting recovery of all containers Sep 09 23:16:46.542880 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.541584 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-167.ec2.internal.18d3ca3521261c25 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-167.ec2.internal,UID:ip-10-0-142-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-167.ec2.internal,},FirstTimestamp:2026-09-09 23:16:46.534442021 +0000 UTC m=+0.347050373,LastTimestamp:2026-09-09 23:16:46.534442021 +0000 UTC m=+0.347050373,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-167.ec2.internal,}" Sep 09 23:16:46.549621 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.549576 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 23:16:46.549710 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.549625 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 23:16:46.554469 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.554452 2559 manager.go:354] Recovery completed Sep 09 23:16:46.558952 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.558936 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rqbjc" Sep 09 23:16:46.559503 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.559492 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:16:46.565271 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.565257 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:16:46.565349 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.565282 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:16:46.565349 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.565304 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:16:46.566233 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.566213 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rqbjc" Sep 09 23:16:46.566345 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.566328 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 23:16:46.566345 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.566343 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 23:16:46.566500 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.566357 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:16:46.566500 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.566362 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:16:46.567177 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.567161 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:16:46.567242 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.567185 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:16:46.567242 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.567196 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:16:46.567242 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.567143 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-167.ec2.internal.18d3ca3522fc83ca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-167.ec2.internal,UID:ip-10-0-142-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-167.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-167.ec2.internal,},FirstTimestamp:2026-09-09 23:16:46.565270474 +0000 UTC m=+0.377878826,LastTimestamp:2026-09-09 23:16:46.565270474 +0000 UTC m=+0.377878826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-167.ec2.internal,}" Sep 09 23:16:46.568475 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.568460 2559 policy_none.go:49] "None policy: Start" Sep 09 23:16:46.568558 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.568478 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 23:16:46.568558 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.568492 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 23:16:46.570428 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.570416 2559 policy_none.go:47] "Start" Sep 09 23:16:46.614213 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.614191 2559 manager.go:337] "Starting Device Plugin manager" Sep 09 23:16:46.614350 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.614231 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 23:16:46.614350 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.614243 2559 server.go:87] "Starting device plugin registration server" Sep 09 23:16:46.614535 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.614523 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 23:16:46.614578 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.614537 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 23:16:46.614731 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.614708 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 23:16:46.614806 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.614789 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 23:16:46.614806 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.614799 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 23:16:46.615238 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.615213 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 23:16:46.615311 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.615273 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:46.636337 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.636314 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 23:16:46.637437 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.637413 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 23:16:46.637516 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.637469 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 23:16:46.637516 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.637490 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 23:16:46.637637 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.637573 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 23:16:46.640243 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.640223 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:16:46.715188 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.715133 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:16:46.715976 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.715962 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:16:46.716045 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.715989 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:16:46.716045 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.716005 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:16:46.716045 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.716031 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.724869 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.724851 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.724935 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.724870 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-167.ec2.internal\": node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:46.738279 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.738257 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal"] Sep 09 23:16:46.738363 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.738327 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:16:46.739562 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.739546 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:16:46.739657 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.739566 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:16:46.739657 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.739576 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:16:46.741754 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.741741 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:16:46.741873 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.741858 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.741930 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.741891 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:16:46.742037 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/420d00517fcec5d34534f0cd4e57dfdc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal\" (UID: \"420d00517fcec5d34534f0cd4e57dfdc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.742094 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/420d00517fcec5d34534f0cd4e57dfdc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal\" (UID: \"420d00517fcec5d34534f0cd4e57dfdc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.742371 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742357 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:16:46.742415 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742383 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:16:46.742415 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742392 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:16:46.742484 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742360 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:16:46.742484 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742455 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:16:46.742484 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.742464 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:16:46.744740 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.744727 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.744795 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.744749 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:16:46.745389 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.745375 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:16:46.745466 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.745398 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:16:46.745466 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.745407 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:16:46.753553 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.753537 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:46.773111 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.773087 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-167.ec2.internal\" not found" node="ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.777281 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.777266 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-167.ec2.internal\" not found" node="ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.842770 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.842747 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/420d00517fcec5d34534f0cd4e57dfdc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal\" (UID: \"420d00517fcec5d34534f0cd4e57dfdc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.842853 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.842783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/420d00517fcec5d34534f0cd4e57dfdc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal\" (UID: \"420d00517fcec5d34534f0cd4e57dfdc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.842853 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.842790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/420d00517fcec5d34534f0cd4e57dfdc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal\" (UID: \"420d00517fcec5d34534f0cd4e57dfdc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.842853 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.842803 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55584c1d8f97229f4aae3d06860d9639-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.ec2.internal\" (UID: \"55584c1d8f97229f4aae3d06860d9639\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.842853 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.842842 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/420d00517fcec5d34534f0cd4e57dfdc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal\" (UID: \"420d00517fcec5d34534f0cd4e57dfdc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.853879 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.853858 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:46.943234 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.943215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55584c1d8f97229f4aae3d06860d9639-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.ec2.internal\" (UID: \"55584c1d8f97229f4aae3d06860d9639\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.943306 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:46.943263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/55584c1d8f97229f4aae3d06860d9639-config\") pod \"kube-apiserver-proxy-ip-10-0-142-167.ec2.internal\" (UID: \"55584c1d8f97229f4aae3d06860d9639\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" Sep 09 23:16:46.954329 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:46.954313 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:47.055147 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:47.055097 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:47.076254 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.076232 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:47.080993 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.080980 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" Sep 09 23:16:47.155711 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:47.155691 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:47.256206 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:47.256190 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:47.356700 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:47.356649 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:47.456929 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:47.456908 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:47.486135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.486114 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 23:16:47.486499 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.486246 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 23:16:47.557233 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:47.557214 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-167.ec2.internal\" not found" Sep 09 23:16:47.569697 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.569667 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 23:11:46 +0000 UTC" deadline="2028-02-04 18:34:15.745535013 +0000 UTC" Sep 09 23:16:47.569697 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.569692 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12307h17m28.175845847s" Sep 09 23:16:47.597856 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.597834 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:16:47.641060 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.641007 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" Sep 09 23:16:47.651709 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.651687 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:16:47.653716 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.653700 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" Sep 09 23:16:47.666082 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.666064 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:16:47.719369 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.719349 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:16:47.806847 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.806818 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:16:47.852897 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:47.852869 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d00517fcec5d34534f0cd4e57dfdc.slice/crio-732073fb49d82aa11f430a28be359518196724d0d3fb9e1b68187e8eb2de401d WatchSource:0}: Error finding container 732073fb49d82aa11f430a28be359518196724d0d3fb9e1b68187e8eb2de401d: Status 404 returned error can't find the container with id 732073fb49d82aa11f430a28be359518196724d0d3fb9e1b68187e8eb2de401d Sep 09 23:16:47.853093 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:47.853073 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55584c1d8f97229f4aae3d06860d9639.slice/crio-e9410fc6f15b8c3d5919811f76067122d7b69d2ec201e021d1f68758a1e49e06 WatchSource:0}: Error finding container e9410fc6f15b8c3d5919811f76067122d7b69d2ec201e021d1f68758a1e49e06: Status 404 returned error can't find the container with id e9410fc6f15b8c3d5919811f76067122d7b69d2ec201e021d1f68758a1e49e06 Sep 09 23:16:47.857207 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:47.857187 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:16:48.526137 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.526102 2559 apiserver.go:52] "Watching apiserver" Sep 09 23:16:48.539288 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.539265 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 23:16:48.540666 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.540641 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5rgz4","kube-system/konnectivity-agent-lb6gg","kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lkfkw","openshift-dns/node-resolver-b2z6r","openshift-image-registry/node-ca-ld9zs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal","openshift-multus/multus-additional-cni-plugins-28zfb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv","openshift-multus/multus-ss8v8","openshift-multus/network-metrics-daemon-ht64b","openshift-network-diagnostics/network-check-target-bdjlj","openshift-network-operator/iptables-alerter-c4m7m","openshift-ovn-kubernetes/ovnkube-node-b9zzg"] Sep 09 23:16:48.541822 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.541803 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 23:16:48.544835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.544818 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.547045 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.547021 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.547839 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.547818 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c5sq8\"" Sep 09 23:16:48.547951 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.547825 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 23:16:48.548089 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.548074 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 23:16:48.548160 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.548103 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 23:16:48.548218 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.548154 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 23:16:48.549751 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.549730 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 23:16:48.549927 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.549908 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wfndt\"" Sep 09 23:16:48.550089 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.550064 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.550246 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.550224 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 23:16:48.550453 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.550429 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:16:48.551443 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-conf-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551543 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551456 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-cni-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551543 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-cnibin\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551543 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551507 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-cni-multus\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551545 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-socket-dir-parent\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551570 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-daemon-config\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-etc-kubernetes\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbzm\" (UniqueName: \"kubernetes.io/projected/11c60a16-cbba-4a3b-b165-66da33d20d8b-kube-api-access-mlbzm\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-hostroot\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551685 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-system-cni-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.551741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-netns\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.552099 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551743 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-cni-bin\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.552099 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-multus-certs\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.552099 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c60a16-cbba-4a3b-b165-66da33d20d8b-cni-binary-copy\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.552099 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-k8s-cni-cncf-io\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.552099 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.551850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-kubelet\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.552836 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.552497 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.553340 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.553318 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 23:16:48.553424 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.553375 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rcnn6\"" Sep 09 23:16:48.553529 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.553511 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 23:16:48.553585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.553569 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 23:16:48.553715 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.553645 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:16:48.553715 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.553660 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 23:16:48.554431 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.554415 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 23:16:48.554513 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.554420 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 23:16:48.554792 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.554776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.555715 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.555694 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fs2hq\"" Sep 09 23:16:48.555948 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.555928 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:16:48.556301 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.556282 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 23:16:48.557884 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.557860 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 23:16:48.557976 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.557916 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bxdbl\"" Sep 09 23:16:48.558175 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.558157 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 23:16:48.558240 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.558225 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.560587 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.560567 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 23:16:48.561294 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.560906 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 23:16:48.561294 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.560959 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 23:16:48.561294 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.560916 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8src2\"" Sep 09 23:16:48.561747 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.561729 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.563537 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.563517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.564439 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.564420 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 23:16:48.564890 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.564866 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sbqmf\"" Sep 09 23:16:48.564890 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.564886 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 23:16:48.565207 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.565181 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 23:16:48.565990 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.565967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.566368 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.566348 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z8ggh\"" Sep 09 23:16:48.566958 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.566936 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 23:16:48.567306 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.567263 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 23:16:48.569302 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.569243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.569302 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.569275 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 23:16:48.569302 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.569301 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 23:16:48.569511 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.569324 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q2xqz\"" Sep 09 23:16:48.569511 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.569300 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:16:48.572965 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.572907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:48.575905 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.573093 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:16:48.576708 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.576485 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:48.576708 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.576565 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:16:48.578962 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.578935 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bgzgv" Sep 09 23:16:48.587650 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.587626 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bgzgv" Sep 09 23:16:48.642545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.642522 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 23:16:48.644461 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.644408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" event={"ID":"55584c1d8f97229f4aae3d06860d9639","Type":"ContainerStarted","Data":"e9410fc6f15b8c3d5919811f76067122d7b69d2ec201e021d1f68758a1e49e06"} Sep 09 23:16:48.646199 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.646165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" event={"ID":"420d00517fcec5d34534f0cd4e57dfdc","Type":"ContainerStarted","Data":"732073fb49d82aa11f430a28be359518196724d0d3fb9e1b68187e8eb2de401d"} Sep 09 23:16:48.652621 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652586 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-log-socket\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.652739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wc4\" (UniqueName: \"kubernetes.io/projected/f0d6b534-13fa-4644-801b-c85e53abb675-kube-api-access-n4wc4\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.652739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbzm\" (UniqueName: \"kubernetes.io/projected/11c60a16-cbba-4a3b-b165-66da33d20d8b-kube-api-access-mlbzm\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.652854 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652756 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44271e39-9e4f-4db6-b6da-ad10fc033a48-iptables-alerter-script\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.652854 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-cni-bin\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.652854 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-ovnkube-script-lib\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.652854 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysctl-d\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652867 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-hostroot\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/44271e39-9e4f-4db6-b6da-ad10fc033a48-kube-api-access-w9mdk\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22846b1d-d458-426a-9a93-0b2c598879c1-ovn-node-metrics-cert\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rkj\" (UniqueName: \"kubernetes.io/projected/22846b1d-d458-426a-9a93-0b2c598879c1-kube-api-access-z2rkj\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-device-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-run-netns\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-hostroot\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.652989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-ovn\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.653055 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-socket-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-cnibin\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-cni-multus\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysconfig\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-host\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653199 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44271e39-9e4f-4db6-b6da-ad10fc033a48-host-slash\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-env-overrides\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653377 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27ef369c-d564-486b-af79-a543ba16dc83-agent-certs\") pod \"konnectivity-agent-lb6gg\" (UID: \"27ef369c-d564-486b-af79-a543ba16dc83\") " pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-netns\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-etc-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653463 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826xg\" (UniqueName: \"kubernetes.io/projected/70c765f6-1f3d-4660-bc36-48293230f6e3-kube-api-access-826xg\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.653563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-registration-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653584 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-sys\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653612 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c60a16-cbba-4a3b-b165-66da33d20d8b-cni-binary-copy\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653646 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a33b8ed-f089-4b62-b83c-23c8db52500b-dbus\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-kubelet\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-modprobe-d\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/633f88c9-63c9-48d4-9ffc-523090d75649-hosts-file\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70c765f6-1f3d-4660-bc36-48293230f6e3-serviceca\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653789 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-cni-multus\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-cnibin\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-netns\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-cni-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653919 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653983 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-cni-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.653987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-systemd\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.654285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-var-lib-kubelet\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-system-cni-dir\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-socket-dir-parent\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-socket-dir-parent\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654250 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-cni-netd\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654318 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-ovnkube-config\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0d6b534-13fa-4644-801b-c85e53abb675-etc-tuned\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654386 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-system-cni-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-multus-certs\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-systemd-units\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654488 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-lib-modules\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-cnibin\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-system-cni-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654564 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-sys-fs\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654593 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-conf-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.655187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-multus-certs\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654630 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-run\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-node-log\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-conf-dir\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-kubernetes\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/633f88c9-63c9-48d4-9ffc-523090d75649-tmp-dir\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654725 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70c765f6-1f3d-4660-bc36-48293230f6e3-host\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74spk\" (UniqueName: \"kubernetes.io/projected/dcfe056a-125c-4ea2-911f-113adcf74363-kube-api-access-74spk\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-daemon-config\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-etc-kubernetes\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a33b8ed-f089-4b62-b83c-23c8db52500b-kubelet-config\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654864 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzth\" (UniqueName: \"kubernetes.io/projected/633f88c9-63c9-48d4-9ffc-523090d75649-kube-api-access-xpzth\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654889 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-etc-kubernetes\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654932 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27ef369c-d564-486b-af79-a543ba16dc83-konnectivity-ca\") pod \"konnectivity-agent-lb6gg\" (UID: \"27ef369c-d564-486b-af79-a543ba16dc83\") " pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c60a16-cbba-4a3b-b165-66da33d20d8b-cni-binary-copy\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656076 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-cni-bin\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.654987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-var-lib-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-cni-bin\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655121 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0d6b534-13fa-4644-801b-c85e53abb675-tmp\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655149 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv2cx\" (UniqueName: \"kubernetes.io/projected/14ee243b-7559-462e-9414-a47ed4df6877-kube-api-access-fv2cx\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-k8s-cni-cncf-io\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-kubelet\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-run-k8s-cni-cncf-io\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd56b\" (UniqueName: \"kubernetes.io/projected/060a36f0-058c-4110-a094-dff084254fb5-kube-api-access-wd56b\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655313 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-slash\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11c60a16-cbba-4a3b-b165-66da33d20d8b-multus-daemon-config\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11c60a16-cbba-4a3b-b165-66da33d20d8b-host-var-lib-kubelet\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-systemd\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655442 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysctl-conf\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.656961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.655467 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-cni-binary-copy\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.659378 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.659346 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 23:16:48.664300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.664279 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbzm\" (UniqueName: \"kubernetes.io/projected/11c60a16-cbba-4a3b-b165-66da33d20d8b-kube-api-access-mlbzm\") pod \"multus-ss8v8\" (UID: \"11c60a16-cbba-4a3b-b165-66da33d20d8b\") " pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.755859 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.755820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44271e39-9e4f-4db6-b6da-ad10fc033a48-iptables-alerter-script\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.755859 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.755855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-cni-bin\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.755873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-ovnkube-script-lib\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.755897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysctl-d\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.755937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/44271e39-9e4f-4db6-b6da-ad10fc033a48-kube-api-access-w9mdk\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.755955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-cni-bin\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.755961 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22846b1d-d458-426a-9a93-0b2c598879c1-ovn-node-metrics-cert\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rkj\" (UniqueName: \"kubernetes.io/projected/22846b1d-d458-426a-9a93-0b2c598879c1-kube-api-access-z2rkj\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-device-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-run-netns\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756111 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysctl-d\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-ovn\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756145 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-socket-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756176 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysconfig\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-host\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-run-netns\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44271e39-9e4f-4db6-b6da-ad10fc033a48-host-slash\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756331 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-env-overrides\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.756352 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:48.756611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-device-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.756920 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.756768 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret podName:1a33b8ed-f089-4b62-b83c-23c8db52500b nodeName:}" failed. No retries permitted until 2026-09-09 23:16:49.256720326 +0000 UTC m=+3.069328670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret") pod "global-pull-secret-syncer-5rgz4" (UID: "1a33b8ed-f089-4b62-b83c-23c8db52500b") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:48.756993 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/44271e39-9e4f-4db6-b6da-ad10fc033a48-iptables-alerter-script\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.757107 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-ovn\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.757253 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.757253 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-ovnkube-script-lib\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.757253 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.756392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.757253 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757214 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysconfig\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.757253 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757232 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-socket-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.757253 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27ef369c-d564-486b-af79-a543ba16dc83-agent-certs\") pod \"konnectivity-agent-lb6gg\" (UID: \"27ef369c-d564-486b-af79-a543ba16dc83\") " pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.757545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757278 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-host\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.757545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757295 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44271e39-9e4f-4db6-b6da-ad10fc033a48-host-slash\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.757545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-etc-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.757545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757475 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826xg\" (UniqueName: \"kubernetes.io/projected/70c765f6-1f3d-4660-bc36-48293230f6e3-kube-api-access-826xg\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.757545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757512 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-registration-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.757545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757540 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-sys\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757579 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a33b8ed-f089-4b62-b83c-23c8db52500b-dbus\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-kubelet\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-modprobe-d\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-env-overrides\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/633f88c9-63c9-48d4-9ffc-523090d75649-hosts-file\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70c765f6-1f3d-4660-bc36-48293230f6e3-serviceca\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757795 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/633f88c9-63c9-48d4-9ffc-523090d75649-hosts-file\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.757905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-etc-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a33b8ed-f089-4b62-b83c-23c8db52500b-dbus\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-modprobe-d\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.757824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-kubelet\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-registration-dir\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:48.758333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758390 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70c765f6-1f3d-4660-bc36-48293230f6e3-serviceca\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-sys\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758478 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.758494 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758520 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-systemd\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.758570 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs podName:060a36f0-058c-4110-a094-dff084254fb5 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:49.25854027 +0000 UTC m=+3.071148622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs") pod "network-metrics-daemon-ht64b" (UID: "060a36f0-058c-4110-a094-dff084254fb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-var-lib-kubelet\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758658 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-system-cni-dir\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758675 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-systemd\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.758717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-cni-netd\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-var-lib-kubelet\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758753 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-cni-netd\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-system-cni-dir\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-ovnkube-config\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0d6b534-13fa-4644-801b-c85e53abb675-etc-tuned\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-systemd-units\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758918 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-lib-modules\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-cnibin\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.758980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759023 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-sys-fs\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-run\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-node-log\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759117 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-lib-modules\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-kubernetes\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.759180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/633f88c9-63c9-48d4-9ffc-523090d75649-tmp-dir\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-kubernetes\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759496 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/633f88c9-63c9-48d4-9ffc-523090d75649-tmp-dir\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759586 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-run\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-systemd-units\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-cnibin\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dcfe056a-125c-4ea2-911f-113adcf74363-sys-fs\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-node-log\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759765 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22846b1d-d458-426a-9a93-0b2c598879c1-ovnkube-config\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70c765f6-1f3d-4660-bc36-48293230f6e3-host\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759829 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70c765f6-1f3d-4660-bc36-48293230f6e3-host\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74spk\" (UniqueName: \"kubernetes.io/projected/dcfe056a-125c-4ea2-911f-113adcf74363-kube-api-access-74spk\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a33b8ed-f089-4b62-b83c-23c8db52500b-kubelet-config\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzth\" (UniqueName: \"kubernetes.io/projected/633f88c9-63c9-48d4-9ffc-523090d75649-kube-api-access-xpzth\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.759965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27ef369c-d564-486b-af79-a543ba16dc83-konnectivity-ca\") pod \"konnectivity-agent-lb6gg\" (UID: \"27ef369c-d564-486b-af79-a543ba16dc83\") " pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.760026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-var-lib-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0d6b534-13fa-4644-801b-c85e53abb675-tmp\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv2cx\" (UniqueName: \"kubernetes.io/projected/14ee243b-7559-462e-9414-a47ed4df6877-kube-api-access-fv2cx\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd56b\" (UniqueName: \"kubernetes.io/projected/060a36f0-058c-4110-a094-dff084254fb5-kube-api-access-wd56b\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-slash\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-systemd\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysctl-conf\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-cni-binary-copy\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14ee243b-7559-462e-9414-a47ed4df6877-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760314 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-log-socket\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wc4\" (UniqueName: \"kubernetes.io/projected/f0d6b534-13fa-4644-801b-c85e53abb675-kube-api-access-n4wc4\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a33b8ed-f089-4b62-b83c-23c8db52500b-kubelet-config\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760608 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-log-socket\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.760835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760705 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0d6b534-13fa-4644-801b-c85e53abb675-etc-sysctl-conf\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.761527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.760996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14ee243b-7559-462e-9414-a47ed4df6877-cni-binary-copy\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.761527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.761043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22846b1d-d458-426a-9a93-0b2c598879c1-ovn-node-metrics-cert\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.761527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.761212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-var-lib-openvswitch\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.761527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.761269 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.761527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.761321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27ef369c-d564-486b-af79-a543ba16dc83-konnectivity-ca\") pod \"konnectivity-agent-lb6gg\" (UID: \"27ef369c-d564-486b-af79-a543ba16dc83\") " pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.761527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.761363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-host-slash\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.761527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.761430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22846b1d-d458-426a-9a93-0b2c598879c1-run-systemd\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.764047 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.763959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0d6b534-13fa-4644-801b-c85e53abb675-tmp\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.764198 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.764179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27ef369c-d564-486b-af79-a543ba16dc83-agent-certs\") pod \"konnectivity-agent-lb6gg\" (UID: \"27ef369c-d564-486b-af79-a543ba16dc83\") " pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.765345 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.765051 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0d6b534-13fa-4644-801b-c85e53abb675-etc-tuned\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.767507 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.767474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/44271e39-9e4f-4db6-b6da-ad10fc033a48-kube-api-access-w9mdk\") pod \"iptables-alerter-c4m7m\" (UID: \"44271e39-9e4f-4db6-b6da-ad10fc033a48\") " pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.768112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.768087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rkj\" (UniqueName: \"kubernetes.io/projected/22846b1d-d458-426a-9a93-0b2c598879c1-kube-api-access-z2rkj\") pod \"ovnkube-node-b9zzg\" (UID: \"22846b1d-d458-426a-9a93-0b2c598879c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.768946 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.768926 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:16:48.769076 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.769061 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:16:48.769194 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.769181 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9bsjr for pod openshift-network-diagnostics/network-check-target-bdjlj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:48.769363 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:48.769348 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr podName:81c0e958-8a2c-47cc-804c-50e383fe3b03 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:49.269327488 +0000 UTC m=+3.081935830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9bsjr" (UniqueName: "kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr") pod "network-check-target-bdjlj" (UID: "81c0e958-8a2c-47cc-804c-50e383fe3b03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:48.769471 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.769131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-826xg\" (UniqueName: \"kubernetes.io/projected/70c765f6-1f3d-4660-bc36-48293230f6e3-kube-api-access-826xg\") pod \"node-ca-ld9zs\" (UID: \"70c765f6-1f3d-4660-bc36-48293230f6e3\") " pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.770470 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.770434 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv2cx\" (UniqueName: \"kubernetes.io/projected/14ee243b-7559-462e-9414-a47ed4df6877-kube-api-access-fv2cx\") pod \"multus-additional-cni-plugins-28zfb\" (UID: \"14ee243b-7559-462e-9414-a47ed4df6877\") " pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.771592 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.771565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzth\" (UniqueName: \"kubernetes.io/projected/633f88c9-63c9-48d4-9ffc-523090d75649-kube-api-access-xpzth\") pod \"node-resolver-b2z6r\" (UID: \"633f88c9-63c9-48d4-9ffc-523090d75649\") " pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.772147 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.772127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd56b\" (UniqueName: \"kubernetes.io/projected/060a36f0-058c-4110-a094-dff084254fb5-kube-api-access-wd56b\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:48.772841 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.772818 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wc4\" (UniqueName: \"kubernetes.io/projected/f0d6b534-13fa-4644-801b-c85e53abb675-kube-api-access-n4wc4\") pod \"tuned-lkfkw\" (UID: \"f0d6b534-13fa-4644-801b-c85e53abb675\") " pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.772954 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.772936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74spk\" (UniqueName: \"kubernetes.io/projected/dcfe056a-125c-4ea2-911f-113adcf74363-kube-api-access-74spk\") pod \"aws-ebs-csi-driver-node-pmpsv\" (UID: \"dcfe056a-125c-4ea2-911f-113adcf74363\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.856839 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.856757 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ss8v8" Sep 09 23:16:48.866354 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.866176 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4m7m" Sep 09 23:16:48.873555 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.873290 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c60a16_cbba_4a3b_b165_66da33d20d8b.slice/crio-f279475cba611e06499b363fd4d2e4af0e18bba4e9ff8f80c318fb108619bc63 WatchSource:0}: Error finding container f279475cba611e06499b363fd4d2e4af0e18bba4e9ff8f80c318fb108619bc63: Status 404 returned error can't find the container with id f279475cba611e06499b363fd4d2e4af0e18bba4e9ff8f80c318fb108619bc63 Sep 09 23:16:48.877799 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.877774 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44271e39_9e4f_4db6_b6da_ad10fc033a48.slice/crio-3efa70075dae8bcb4e6cabbba6dd633d45ae787fa16dccf862e126445e977abd WatchSource:0}: Error finding container 3efa70075dae8bcb4e6cabbba6dd633d45ae787fa16dccf862e126445e977abd: Status 404 returned error can't find the container with id 3efa70075dae8bcb4e6cabbba6dd633d45ae787fa16dccf862e126445e977abd Sep 09 23:16:48.881169 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.880371 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:16:48.883614 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.883418 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:16:48.889785 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.889765 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" Sep 09 23:16:48.892841 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.892818 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22846b1d_d458_426a_9a93_0b2c598879c1.slice/crio-417b924b0eadfe37faab84efadf47969aca9a2f40fd7ef33ac10281181e82879 WatchSource:0}: Error finding container 417b924b0eadfe37faab84efadf47969aca9a2f40fd7ef33ac10281181e82879: Status 404 returned error can't find the container with id 417b924b0eadfe37faab84efadf47969aca9a2f40fd7ef33ac10281181e82879 Sep 09 23:16:48.896650 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.896631 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b2z6r" Sep 09 23:16:48.903948 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.903924 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d6b534_13fa_4644_801b_c85e53abb675.slice/crio-cab44814f1e57f263eef4ab59ce8d47e605e8071483e072cf9f36b51f6b13d38 WatchSource:0}: Error finding container cab44814f1e57f263eef4ab59ce8d47e605e8071483e072cf9f36b51f6b13d38: Status 404 returned error can't find the container with id cab44814f1e57f263eef4ab59ce8d47e605e8071483e072cf9f36b51f6b13d38 Sep 09 23:16:48.905193 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.905176 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ld9zs" Sep 09 23:16:48.915017 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.914996 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" Sep 09 23:16:48.919410 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.919354 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c765f6_1f3d_4660_bc36_48293230f6e3.slice/crio-f258a46ae04154142ecbecc12ba39858aa8c408ce6f7bc446e888770036248ad WatchSource:0}: Error finding container f258a46ae04154142ecbecc12ba39858aa8c408ce6f7bc446e888770036248ad: Status 404 returned error can't find the container with id f258a46ae04154142ecbecc12ba39858aa8c408ce6f7bc446e888770036248ad Sep 09 23:16:48.923303 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.923282 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-28zfb" Sep 09 23:16:48.926292 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.926266 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfe056a_125c_4ea2_911f_113adcf74363.slice/crio-2fbd0c7806677d4f2889e449f26f714605cc03e639776b6029bdc4a0f628762b WatchSource:0}: Error finding container 2fbd0c7806677d4f2889e449f26f714605cc03e639776b6029bdc4a0f628762b: Status 404 returned error can't find the container with id 2fbd0c7806677d4f2889e449f26f714605cc03e639776b6029bdc4a0f628762b Sep 09 23:16:48.931237 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:48.931066 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:16:48.934517 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.934493 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ee243b_7559_462e_9414_a47ed4df6877.slice/crio-53f777a4c247bfa7ef1e9764e28e1e058170ccd831e19bf44170233a9eef92c9 WatchSource:0}: Error finding container 53f777a4c247bfa7ef1e9764e28e1e058170ccd831e19bf44170233a9eef92c9: Status 404 returned error can't find the container with id 53f777a4c247bfa7ef1e9764e28e1e058170ccd831e19bf44170233a9eef92c9 Sep 09 23:16:48.943063 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:16:48.943026 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ef369c_d564_486b_af79_a543ba16dc83.slice/crio-f6e77a5daa8dad1a82a311db614b959b7657ec23aff546f1436eb0e0a6aa119e WatchSource:0}: Error finding container f6e77a5daa8dad1a82a311db614b959b7657ec23aff546f1436eb0e0a6aa119e: Status 404 returned error can't find the container with id f6e77a5daa8dad1a82a311db614b959b7657ec23aff546f1436eb0e0a6aa119e Sep 09 23:16:49.264271 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.264190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:49.264430 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.264296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:49.264430 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.264370 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:49.264430 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.264415 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:49.264614 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.264458 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs podName:060a36f0-058c-4110-a094-dff084254fb5 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:50.264435769 +0000 UTC m=+4.077044115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs") pod "network-metrics-daemon-ht64b" (UID: "060a36f0-058c-4110-a094-dff084254fb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:49.264614 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.264479 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret podName:1a33b8ed-f089-4b62-b83c-23c8db52500b nodeName:}" failed. No retries permitted until 2026-09-09 23:16:50.264468906 +0000 UTC m=+4.077077249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret") pod "global-pull-secret-syncer-5rgz4" (UID: "1a33b8ed-f089-4b62-b83c-23c8db52500b") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:49.364870 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.364837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:49.365049 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.364992 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:16:49.365049 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.365014 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:16:49.365049 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.365024 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9bsjr for pod openshift-network-diagnostics/network-check-target-bdjlj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:49.365231 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:49.365072 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr podName:81c0e958-8a2c-47cc-804c-50e383fe3b03 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:50.365058968 +0000 UTC m=+4.177667309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bsjr" (UniqueName: "kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr") pod "network-check-target-bdjlj" (UID: "81c0e958-8a2c-47cc-804c-50e383fe3b03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:49.591354 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.591248 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:11:48 +0000 UTC" deadline="2028-03-28 10:01:17.107826 +0000 UTC" Sep 09 23:16:49.591354 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.591306 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13570h44m27.516525402s" Sep 09 23:16:49.650078 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.649967 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lb6gg" event={"ID":"27ef369c-d564-486b-af79-a543ba16dc83","Type":"ContainerStarted","Data":"f6e77a5daa8dad1a82a311db614b959b7657ec23aff546f1436eb0e0a6aa119e"} Sep 09 23:16:49.651478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.651410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerStarted","Data":"53f777a4c247bfa7ef1e9764e28e1e058170ccd831e19bf44170233a9eef92c9"} Sep 09 23:16:49.653084 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.653027 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ld9zs" event={"ID":"70c765f6-1f3d-4660-bc36-48293230f6e3","Type":"ContainerStarted","Data":"f258a46ae04154142ecbecc12ba39858aa8c408ce6f7bc446e888770036248ad"} Sep 09 23:16:49.655601 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.655540 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" event={"ID":"f0d6b534-13fa-4644-801b-c85e53abb675","Type":"ContainerStarted","Data":"cab44814f1e57f263eef4ab59ce8d47e605e8071483e072cf9f36b51f6b13d38"} Sep 09 23:16:49.657279 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.657248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"417b924b0eadfe37faab84efadf47969aca9a2f40fd7ef33ac10281181e82879"} Sep 09 23:16:49.659355 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.659230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4m7m" event={"ID":"44271e39-9e4f-4db6-b6da-ad10fc033a48","Type":"ContainerStarted","Data":"3efa70075dae8bcb4e6cabbba6dd633d45ae787fa16dccf862e126445e977abd"} Sep 09 23:16:49.661454 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.661433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" event={"ID":"dcfe056a-125c-4ea2-911f-113adcf74363","Type":"ContainerStarted","Data":"2fbd0c7806677d4f2889e449f26f714605cc03e639776b6029bdc4a0f628762b"} Sep 09 23:16:49.663056 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.663031 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b2z6r" event={"ID":"633f88c9-63c9-48d4-9ffc-523090d75649","Type":"ContainerStarted","Data":"4580a25bb5119e1228041ad7b149a87b2e787584b919e4ffddefa4e46a66d96e"} Sep 09 23:16:49.664446 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:49.664411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ss8v8" event={"ID":"11c60a16-cbba-4a3b-b165-66da33d20d8b","Type":"ContainerStarted","Data":"f279475cba611e06499b363fd4d2e4af0e18bba4e9ff8f80c318fb108619bc63"} Sep 09 23:16:50.273026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.272201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:50.273026 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.272262 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:50.273026 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.272419 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:50.273026 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.272486 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs podName:060a36f0-058c-4110-a094-dff084254fb5 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:52.272465842 +0000 UTC m=+6.085074189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs") pod "network-metrics-daemon-ht64b" (UID: "060a36f0-058c-4110-a094-dff084254fb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:50.273026 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.272933 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:50.273026 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.272988 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret podName:1a33b8ed-f089-4b62-b83c-23c8db52500b nodeName:}" failed. No retries permitted until 2026-09-09 23:16:52.272971922 +0000 UTC m=+6.085580270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret") pod "global-pull-secret-syncer-5rgz4" (UID: "1a33b8ed-f089-4b62-b83c-23c8db52500b") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:50.374348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.373649 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:50.374348 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.373875 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:16:50.374348 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.373899 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:16:50.374348 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.373911 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9bsjr for pod openshift-network-diagnostics/network-check-target-bdjlj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:50.374348 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.373972 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr podName:81c0e958-8a2c-47cc-804c-50e383fe3b03 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:52.373953173 +0000 UTC m=+6.186561520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bsjr" (UniqueName: "kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr") pod "network-check-target-bdjlj" (UID: "81c0e958-8a2c-47cc-804c-50e383fe3b03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:50.592468 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.592250 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:11:48 +0000 UTC" deadline="2028-03-25 08:01:11.550859924 +0000 UTC" Sep 09 23:16:50.592881 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.592471 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13496h44m20.958394951s" Sep 09 23:16:50.640067 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.640040 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:50.640369 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.640190 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:16:50.640369 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.640274 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:50.640554 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.640387 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:16:50.640554 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:50.640466 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:50.640683 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:50.640562 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:16:51.677164 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:51.677114 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" event={"ID":"55584c1d8f97229f4aae3d06860d9639","Type":"ContainerStarted","Data":"5d8a4f294e638c8b0555b5f72a54bdf3b87b493ed70c29a1abb5390977724ff4"} Sep 09 23:16:51.686156 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:51.686125 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" event={"ID":"420d00517fcec5d34534f0cd4e57dfdc","Type":"ContainerStarted","Data":"f1822445b4c162eb4cb331133c6bc1a83fa0e929bb2d22f1b3e4d9e57ae2631b"} Sep 09 23:16:51.710097 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:51.710042 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-167.ec2.internal" podStartSLOduration=3.710023079 podStartE2EDuration="3.710023079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:16:51.694601289 +0000 UTC m=+5.507209655" watchObservedRunningTime="2026-09-09 23:16:51.710023079 +0000 UTC m=+5.522631445" Sep 09 23:16:52.290887 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:52.290127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:52.290887 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:52.290184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:52.290887 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.290311 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:52.290887 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.290373 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs podName:060a36f0-058c-4110-a094-dff084254fb5 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:56.290353254 +0000 UTC m=+10.102961613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs") pod "network-metrics-daemon-ht64b" (UID: "060a36f0-058c-4110-a094-dff084254fb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:52.290887 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.290810 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:52.290887 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.290862 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret podName:1a33b8ed-f089-4b62-b83c-23c8db52500b nodeName:}" failed. No retries permitted until 2026-09-09 23:16:56.290845395 +0000 UTC m=+10.103453747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret") pod "global-pull-secret-syncer-5rgz4" (UID: "1a33b8ed-f089-4b62-b83c-23c8db52500b") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:52.391343 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:52.390544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:52.391343 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.390786 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:16:52.391343 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.390808 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:16:52.391343 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.390820 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9bsjr for pod openshift-network-diagnostics/network-check-target-bdjlj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:52.391343 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.390880 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr podName:81c0e958-8a2c-47cc-804c-50e383fe3b03 nodeName:}" failed. No retries permitted until 2026-09-09 23:16:56.390862294 +0000 UTC m=+10.203470644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bsjr" (UniqueName: "kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr") pod "network-check-target-bdjlj" (UID: "81c0e958-8a2c-47cc-804c-50e383fe3b03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:52.643957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:52.642923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:52.643957 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.643131 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:16:52.643957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:52.643599 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:52.643957 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.643715 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:16:52.643957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:52.643779 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:52.643957 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:52.643862 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:16:53.693548 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:53.693514 2559 generic.go:356] "Generic (PLEG): container finished" podID="420d00517fcec5d34534f0cd4e57dfdc" containerID="f1822445b4c162eb4cb331133c6bc1a83fa0e929bb2d22f1b3e4d9e57ae2631b" exitCode=0 Sep 09 23:16:53.693949 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:53.693568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" event={"ID":"420d00517fcec5d34534f0cd4e57dfdc","Type":"ContainerDied","Data":"f1822445b4c162eb4cb331133c6bc1a83fa0e929bb2d22f1b3e4d9e57ae2631b"} Sep 09 23:16:54.638828 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:54.638312 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:54.638828 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:54.638344 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:54.638828 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:54.638312 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:54.638828 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:54.638455 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:16:54.638828 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:54.638526 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:16:54.638828 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:54.638615 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:16:56.320298 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:56.320128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:56.320298 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:56.320232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:56.320298 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.320294 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:56.320907 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.320339 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:56.320907 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.320368 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs podName:060a36f0-058c-4110-a094-dff084254fb5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:04.320346176 +0000 UTC m=+18.132954525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs") pod "network-metrics-daemon-ht64b" (UID: "060a36f0-058c-4110-a094-dff084254fb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:16:56.320907 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.320391 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret podName:1a33b8ed-f089-4b62-b83c-23c8db52500b nodeName:}" failed. No retries permitted until 2026-09-09 23:17:04.320380645 +0000 UTC m=+18.132988985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret") pod "global-pull-secret-syncer-5rgz4" (UID: "1a33b8ed-f089-4b62-b83c-23c8db52500b") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:16:56.421549 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:56.421511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:56.421729 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.421703 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:16:56.421729 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.421725 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:16:56.421852 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.421735 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9bsjr for pod openshift-network-diagnostics/network-check-target-bdjlj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:56.421852 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.421782 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr podName:81c0e958-8a2c-47cc-804c-50e383fe3b03 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:04.421769372 +0000 UTC m=+18.234377715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bsjr" (UniqueName: "kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr") pod "network-check-target-bdjlj" (UID: "81c0e958-8a2c-47cc-804c-50e383fe3b03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:16:56.641798 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:56.641729 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:56.641949 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:56.641818 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:56.641949 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:56.641934 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:56.642039 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.641944 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:16:56.642089 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.642036 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:16:56.642136 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:56.642094 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:16:58.644555 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:58.644284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:16:58.644555 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:58.644461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:16:58.644555 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:58.644470 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:16:58.645246 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:16:58.644614 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:16:58.645246 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:58.644696 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:16:58.645246 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:16:58.644757 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:00.640926 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:00.640889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:00.640926 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:00.640923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:00.641436 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:00.640889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:00.641436 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:00.641001 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:00.641436 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:00.641072 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:00.641436 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:00.641155 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:02.641450 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:02.641419 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:02.641450 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:02.641451 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:02.642010 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:02.641545 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:02.642010 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:02.641585 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:02.642010 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:02.641697 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:02.642010 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:02.641784 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:04.375675 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:04.375639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:04.375675 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:04.375681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:04.376161 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.375782 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:17:04.376161 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.375784 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:17:04.376161 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.375846 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret podName:1a33b8ed-f089-4b62-b83c-23c8db52500b nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.375828049 +0000 UTC m=+34.188436398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret") pod "global-pull-secret-syncer-5rgz4" (UID: "1a33b8ed-f089-4b62-b83c-23c8db52500b") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:17:04.376161 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.375865 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs podName:060a36f0-058c-4110-a094-dff084254fb5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.37585478 +0000 UTC m=+34.188463119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs") pod "network-metrics-daemon-ht64b" (UID: "060a36f0-058c-4110-a094-dff084254fb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:17:04.476676 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:04.476630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:04.476853 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.476826 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:17:04.476912 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.476857 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:17:04.476912 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.476872 2559 projected.go:196] Error preparing data for projected volume kube-api-access-9bsjr for pod openshift-network-diagnostics/network-check-target-bdjlj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:17:04.476976 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.476935 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr podName:81c0e958-8a2c-47cc-804c-50e383fe3b03 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.476917515 +0000 UTC m=+34.289525867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bsjr" (UniqueName: "kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr") pod "network-check-target-bdjlj" (UID: "81c0e958-8a2c-47cc-804c-50e383fe3b03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:17:04.637905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:04.637830 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:04.638056 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:04.637840 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:04.638056 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.637948 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:04.638056 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:04.637845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:04.638056 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.638005 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:04.638240 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:04.638121 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:06.638649 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:06.638618 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:06.638991 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:06.638723 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:06.638991 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:06.638734 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:06.638991 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:06.638765 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:06.638991 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:06.638833 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:06.638991 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:06.638886 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:07.720352 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.720125 2559 generic.go:356] "Generic (PLEG): container finished" podID="14ee243b-7559-462e-9414-a47ed4df6877" containerID="a522f36dec3b2a23e56bc250d8a11ee81978cd23986c6b07a38ad2a601fad5d9" exitCode=0 Sep 09 23:17:07.721018 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.720212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerDied","Data":"a522f36dec3b2a23e56bc250d8a11ee81978cd23986c6b07a38ad2a601fad5d9"} Sep 09 23:17:07.722041 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.721991 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ld9zs" event={"ID":"70c765f6-1f3d-4660-bc36-48293230f6e3","Type":"ContainerStarted","Data":"1c1422e0d69e2e595d40cf4149c6fc80ee1d5276bc10e2975c79412da8b8255c"} Sep 09 23:17:07.723510 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.723488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" event={"ID":"f0d6b534-13fa-4644-801b-c85e53abb675","Type":"ContainerStarted","Data":"6f8bb4677613f7e8c88855f11ab1da26f5150bae01ac9550242371fb320a09e4"} Sep 09 23:17:07.726210 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726193 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:17:07.726535 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726515 2559 generic.go:356] "Generic (PLEG): container finished" podID="22846b1d-d458-426a-9a93-0b2c598879c1" containerID="4cb4330fb6e9a98ee7ff5e1fd8f1fd04573fc1101fc2a0b4b92131e531005036" exitCode=1 Sep 09 23:17:07.726614 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726580 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"d194c27c21b79d8615ff9b567d7c35825bbb9ab15fbe8617ba3bca8569e07ce1"} Sep 09 23:17:07.726683 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726643 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"5e5c79c24d018a9690084abc31cd5393d468aa406b5fcd4544642e799e3eb3c8"} Sep 09 23:17:07.726733 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726685 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"68544a08ab9b3f1fc80ec466889597212d7f7df1bd45930244b45e065e4efd82"} Sep 09 23:17:07.726733 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"38553803ee272c7c99409568e8515ba283e8f78ddaba5a3347acb6edd5bb8c2a"} Sep 09 23:17:07.726818 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerDied","Data":"4cb4330fb6e9a98ee7ff5e1fd8f1fd04573fc1101fc2a0b4b92131e531005036"} Sep 09 23:17:07.726818 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.726776 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"4ef194f24340f4f689078fb9ccf828b26ae16310540b2619c633b218c8110b74"} Sep 09 23:17:07.728004 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.727983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" event={"ID":"dcfe056a-125c-4ea2-911f-113adcf74363","Type":"ContainerStarted","Data":"8d0ef4a1dab5ed3098c703765ceaca730def78ed32d8e48fa408cc61418fdf43"} Sep 09 23:17:07.729288 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.729267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b2z6r" event={"ID":"633f88c9-63c9-48d4-9ffc-523090d75649","Type":"ContainerStarted","Data":"499b75aa91ccd473a3ff4ebb4a65d4e86f3a26337081953db89e90b8b23175bb"} Sep 09 23:17:07.730636 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.730616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ss8v8" event={"ID":"11c60a16-cbba-4a3b-b165-66da33d20d8b","Type":"ContainerStarted","Data":"e8db8a5d53ceb4c36dbc545f3669b0d1013a68fff768f46519b1b4fa6696414c"} Sep 09 23:17:07.732343 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.732325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" event={"ID":"420d00517fcec5d34534f0cd4e57dfdc","Type":"ContainerStarted","Data":"bde7537f490fcb6ad5ffa41726dd7cdfe68789beb71dd6a28b22fdb323e6b36a"} Sep 09 23:17:07.733629 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.733610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lb6gg" event={"ID":"27ef369c-d564-486b-af79-a543ba16dc83","Type":"ContainerStarted","Data":"c6ed7f8489bd5253b27bb9927573a229610f543ce48f0c8de795586e362dbd0c"} Sep 09 23:17:07.751583 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.751542 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b2z6r" podStartSLOduration=3.150693355 podStartE2EDuration="20.751530007s" totalImagesPullingTime="17.600836652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:07.751474805 +0000 UTC m=+21.564083168" watchObservedRunningTime="2026-09-09 23:17:07.751530007 +0000 UTC m=+21.564138368" Sep 09 23:17:07.779098 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.779053 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lkfkw" podStartSLOduration=2.772310727 podStartE2EDuration="20.779041913s" totalImagesPullingTime="18.006731186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:07.766232088 +0000 UTC m=+21.578840453" watchObservedRunningTime="2026-09-09 23:17:07.779041913 +0000 UTC m=+21.591650276" Sep 09 23:17:07.779226 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.779203 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ld9zs" podStartSLOduration=3.18740714 podStartE2EDuration="20.779197083s" totalImagesPullingTime="17.591789943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:07.778873665 +0000 UTC m=+21.591482027" watchObservedRunningTime="2026-09-09 23:17:07.779197083 +0000 UTC m=+21.591805448" Sep 09 23:17:07.802372 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.802337 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lb6gg" podStartSLOduration=3.232879864 podStartE2EDuration="20.802326042s" totalImagesPullingTime="17.569446178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:07.802130329 +0000 UTC m=+21.614738690" watchObservedRunningTime="2026-09-09 23:17:07.802326042 +0000 UTC m=+21.614934404" Sep 09 23:17:07.836954 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.836916 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-167.ec2.internal" podStartSLOduration=19.836904924 podStartE2EDuration="19.836904924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:07.818575537 +0000 UTC m=+21.631183899" watchObservedRunningTime="2026-09-09 23:17:07.836904924 +0000 UTC m=+21.649513314" Sep 09 23:17:07.837081 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:07.837063 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ss8v8" podStartSLOduration=2.28783352 podStartE2EDuration="20.837059496s" totalImagesPullingTime="18.549225976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:07.836819748 +0000 UTC m=+21.649428109" watchObservedRunningTime="2026-09-09 23:17:07.837059496 +0000 UTC m=+21.649667858" Sep 09 23:17:08.502235 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.502051 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 23:17:08.628388 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.628190 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T23:17:08.502233203Z","UUID":"0ae5f924-29a3-4abb-ae86-47541a49355a","Handler":null,"Name":"","Endpoint":""} Sep 09 23:17:08.630134 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.630109 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 23:17:08.630262 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.630166 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 23:17:08.637762 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.637739 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:08.637895 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.637743 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:08.637895 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.637875 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:08.638005 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:08.637872 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:08.638005 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:08.637987 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:08.638190 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:08.638163 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:08.737146 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.737110 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4m7m" event={"ID":"44271e39-9e4f-4db6-b6da-ad10fc033a48","Type":"ContainerStarted","Data":"d534d1f766ab3c355945e45e0bcbc64809339c15ad119b1047cc8d299d7a9daf"} Sep 09 23:17:08.739486 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.739450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" event={"ID":"dcfe056a-125c-4ea2-911f-113adcf74363","Type":"ContainerStarted","Data":"df143262fb114655b499d984ddbf0b3489493c3564c338616522a1d0b3508406"} Sep 09 23:17:08.751019 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:08.750979 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c4m7m" podStartSLOduration=4.119408762 podStartE2EDuration="21.750966557s" totalImagesPullingTime="17.631557795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:08.750944856 +0000 UTC m=+22.563553230" watchObservedRunningTime="2026-09-09 23:17:08.750966557 +0000 UTC m=+22.563574919" Sep 09 23:17:09.745272 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:09.745248 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:17:09.745789 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:09.745755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"b72703bc130dd5b395bef7f560538334581f6b87e34eaafff7769996f4fa20d6"} Sep 09 23:17:10.638189 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:10.638147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:10.638467 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:10.638147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:10.638467 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:10.638262 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:10.638467 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:10.638371 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:10.638467 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:10.638159 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:10.638704 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:10.638466 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:10.750846 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:10.750763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" event={"ID":"dcfe056a-125c-4ea2-911f-113adcf74363","Type":"ContainerStarted","Data":"bd6217b4c1e46fcbd78a89c55f4905e93613da98209a8b76f7bcb3d6cfd5b335"} Sep 09 23:17:10.767979 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:10.767936 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmpsv" podStartSLOduration=3.502613545 podStartE2EDuration="23.767922561s" totalImagesPullingTime="20.265309016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:10.767448442 +0000 UTC m=+24.580056796" watchObservedRunningTime="2026-09-09 23:17:10.767922561 +0000 UTC m=+24.580530923" Sep 09 23:17:12.429244 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:12.429209 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:17:12.429841 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:12.429819 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:17:12.638087 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:12.638058 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:12.638252 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:12.638066 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:12.638252 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:12.638169 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:12.638373 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:12.638278 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:12.638373 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:12.638061 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:12.638465 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:12.638411 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:12.754654 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:12.754635 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:17:12.755155 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:12.755090 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lb6gg" Sep 09 23:17:13.758318 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:13.758155 2559 generic.go:356] "Generic (PLEG): container finished" podID="14ee243b-7559-462e-9414-a47ed4df6877" containerID="804d255a30cc832fa294f6c7ab086c0831cdcf6c0e34535fcd2671bdebb89c43" exitCode=0 Sep 09 23:17:13.758802 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:13.758237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerDied","Data":"804d255a30cc832fa294f6c7ab086c0831cdcf6c0e34535fcd2671bdebb89c43"} Sep 09 23:17:13.762195 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:13.762177 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:17:13.762571 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:13.762554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"eae09278fbb3771f0a06330d29e9fca397b1bba54b2d868cca6ab47ecc8852c2"} Sep 09 23:17:13.762777 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:13.762760 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:17:13.762914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:13.762899 2559 scope.go:117] "RemoveContainer" containerID="4cb4330fb6e9a98ee7ff5e1fd8f1fd04573fc1101fc2a0b4b92131e531005036" Sep 09 23:17:13.782154 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:13.782133 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:17:14.638289 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.637922 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:14.638289 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.637952 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:14.638289 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:14.638037 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:14.638289 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:14.638109 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:14.638289 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.638171 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:14.638289 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:14.638253 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:14.641412 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.641390 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rgz4"] Sep 09 23:17:14.642117 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.642097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdjlj"] Sep 09 23:17:14.643009 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.642946 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ht64b"] Sep 09 23:17:14.766999 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.766928 2559 generic.go:356] "Generic (PLEG): container finished" podID="14ee243b-7559-462e-9414-a47ed4df6877" containerID="28443523cba74142154737901d64d2ab3967857dd7c5f029b0bc52b25a9c1c27" exitCode=0 Sep 09 23:17:14.767739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.767016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerDied","Data":"28443523cba74142154737901d64d2ab3967857dd7c5f029b0bc52b25a9c1c27"} Sep 09 23:17:14.771178 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.771158 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:17:14.771560 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.771542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" event={"ID":"22846b1d-d458-426a-9a93-0b2c598879c1","Type":"ContainerStarted","Data":"9dcb9548a310c1b3cb9d4955346a48f565021a6360d2059c45ef6dd9727ab324"} Sep 09 23:17:14.771845 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.771827 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:17:14.771845 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.771838 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:14.771987 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:14.771931 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:14.771987 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.771967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:14.772095 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:14.772068 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:14.772133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.772104 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:14.772198 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.772184 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:17:14.772235 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:14.772193 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:14.786559 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.786542 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:17:14.815603 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:14.815556 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" podStartSLOduration=9.755954477 podStartE2EDuration="27.815542506s" totalImagesPullingTime="18.059588029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:14.814103299 +0000 UTC m=+28.626711661" watchObservedRunningTime="2026-09-09 23:17:14.815542506 +0000 UTC m=+28.628150870" Sep 09 23:17:15.776469 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:15.776436 2559 generic.go:356] "Generic (PLEG): container finished" podID="14ee243b-7559-462e-9414-a47ed4df6877" containerID="6c2a2b273706624c2ef8663db7a4a75979063e618122af984cce325e82bfb988" exitCode=0 Sep 09 23:17:15.776852 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:15.776522 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerDied","Data":"6c2a2b273706624c2ef8663db7a4a75979063e618122af984cce325e82bfb988"} Sep 09 23:17:16.639016 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:16.638821 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:16.639157 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:16.638887 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:16.639157 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:16.639094 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:16.639275 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:16.638916 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:16.639275 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:16.639190 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:16.639376 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:16.639331 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:18.637743 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:18.637707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:18.638198 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:18.637715 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:18.638198 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:18.637810 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:18.638198 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:18.637834 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ht64b" podUID="060a36f0-058c-4110-a094-dff084254fb5" Sep 09 23:17:18.638198 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:18.637900 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rgz4" podUID="1a33b8ed-f089-4b62-b83c-23c8db52500b" Sep 09 23:17:18.638198 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:18.637973 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bdjlj" podUID="81c0e958-8a2c-47cc-804c-50e383fe3b03" Sep 09 23:17:19.938563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.938532 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-167.ec2.internal" event="NodeReady" Sep 09 23:17:19.939013 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.938690 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 23:17:19.975300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.975271 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f9597f89b-ng6z7"] Sep 09 23:17:19.993412 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.993386 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k"] Sep 09 23:17:19.993577 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.993514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:19.997378 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.997349 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r67d2\"" Sep 09 23:17:19.997518 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.997377 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 23:17:19.997518 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.997434 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 23:17:19.997518 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:19.997486 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 23:17:20.001318 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.001289 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65"] Sep 09 23:17:20.001475 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.001373 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:20.006326 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.005967 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.007835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.007571 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 23:17:20.008574 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.008548 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.009764 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.009627 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 09 23:17:20.010553 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.010235 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fwkgt\"" Sep 09 23:17:20.013894 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.013151 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm"] Sep 09 23:17:20.013894 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.013290 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" Sep 09 23:17:20.015767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.015749 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.016018 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.016000 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.016072 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.016058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8w2pg\"" Sep 09 23:17:20.025155 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.024898 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-sktw2"] Sep 09 23:17:20.026225 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.026206 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.029176 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.029160 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 09 23:17:20.029476 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.029458 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.029899 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.029883 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 09 23:17:20.031234 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.031042 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.036794 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.036168 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-k8vtg\"" Sep 09 23:17:20.036936 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.036891 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq"] Sep 09 23:17:20.037024 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.036991 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.040202 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.040183 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 09 23:17:20.040430 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.040412 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.040611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.040465 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xpbsg\"" Sep 09 23:17:20.040611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.040503 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 09 23:17:20.040812 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.040544 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.045859 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.045840 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 09 23:17:20.049155 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.049132 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-f765574b6-292xd"] Sep 09 23:17:20.049274 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.049258 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.054207 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.054187 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 09 23:17:20.054304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.054258 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.054624 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.054468 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pr6dc\"" Sep 09 23:17:20.054914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.054897 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.061899 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.061850 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 09 23:17:20.063154 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.063137 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-wq44d"] Sep 09 23:17:20.063282 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.063265 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.068813 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.068654 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 09 23:17:20.069324 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.069079 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.069324 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.069152 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kwbjl\"" Sep 09 23:17:20.071935 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.071918 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 09 23:17:20.073377 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.073358 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 09 23:17:20.073464 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.073383 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.077100 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.077085 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k"] Sep 09 23:17:20.077173 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.077115 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8jpxs"] Sep 09 23:17:20.077223 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.077211 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.079685 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-ca-trust-extracted\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.079767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-trusted-ca\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.079767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079727 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7abba9dc-6fbe-4927-8298-ddda486d8ed1-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.079767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.079872 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-installation-pull-secrets\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.079872 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-bound-sa-token\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.079872 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztr56\" (UniqueName: \"kubernetes.io/projected/77b46604-88b1-439a-b69d-4689aa67e1f8-kube-api-access-ztr56\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:20.080001 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-certificates\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.080001 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079930 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:20.080001 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.079955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7abba9dc-6fbe-4927-8298-ddda486d8ed1-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.080153 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.080076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-kube-api-access-2xd2z\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.080153 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.080113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4srn\" (UniqueName: \"kubernetes.io/projected/4711a9be-591a-4346-a3ca-15156411555d-kube-api-access-v4srn\") pod \"volume-data-source-validator-695c895bd6-9sz65\" (UID: \"4711a9be-591a-4346-a3ca-15156411555d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" Sep 09 23:17:20.080248 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.080150 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-image-registry-private-configuration\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.080248 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.080179 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2k8\" (UniqueName: \"kubernetes.io/projected/7abba9dc-6fbe-4927-8298-ddda486d8ed1-kube-api-access-ss2k8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.081319 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.081300 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 09 23:17:20.082275 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.082261 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 09 23:17:20.083397 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.083379 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.083488 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.083398 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 09 23:17:20.086823 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.086807 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.088252 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.088236 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p"] Sep 09 23:17:20.088364 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.088343 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.092460 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.092441 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 23:17:20.092717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.092688 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 23:17:20.092717 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.092695 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w7vc5\"" Sep 09 23:17:20.093304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.093283 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zt88c\"" Sep 09 23:17:20.100326 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.100309 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn"] Sep 09 23:17:20.100461 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.100444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.105742 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.105722 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 23:17:20.105843 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.105758 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.105843 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.105828 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.106000 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.105979 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 23:17:20.106097 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.106058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-49x7b\"" Sep 09 23:17:20.111949 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.111926 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 09 23:17:20.118445 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.118422 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-7ghfc"] Sep 09 23:17:20.118541 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.118530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.121159 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.121138 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.121345 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.121325 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 09 23:17:20.121418 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.121343 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-n2zb9\"" Sep 09 23:17:20.121418 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.121333 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.121527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.121329 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 09 23:17:20.140550 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.140526 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg"] Sep 09 23:17:20.140705 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.140687 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" Sep 09 23:17:20.143250 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.143223 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.143346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.143265 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.143346 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.143268 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-92gkc\"" Sep 09 23:17:20.157843 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.157827 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz"] Sep 09 23:17:20.157958 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.157944 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.160727 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.160710 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 09 23:17:20.175122 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.175105 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-sktw2"] Sep 09 23:17:20.175203 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.175135 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq"] Sep 09 23:17:20.175203 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.175150 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-wq44d"] Sep 09 23:17:20.175203 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.175170 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9597f89b-ng6z7"] Sep 09 23:17:20.175203 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.175188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65"] Sep 09 23:17:20.175324 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.175210 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4x4rk"] Sep 09 23:17:20.175324 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.175232 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.177762 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.177718 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 09 23:17:20.177762 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.177740 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 09 23:17:20.177901 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.177793 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 09 23:17:20.177901 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.177800 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 09 23:17:20.180419 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbtn\" (UniqueName: \"kubernetes.io/projected/74e42973-63e2-40bd-ac95-43d33a321f99-kube-api-access-5bbtn\") pod \"managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p\" (UID: \"74e42973-63e2-40bd-ac95-43d33a321f99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.180525 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-certificates\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.180525 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9eea67d0-b1b9-4305-811e-20aa6ea233ed-serving-cert\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.180767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180520 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7abba9dc-6fbe-4927-8298-ddda486d8ed1-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.180767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41109807-ec62-4222-ab2b-43a068961aab-serving-cert\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.180767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f9dc9-9bed-402b-af9f-16b918a054e5-config-volume\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.180767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-kube-api-access-2xd2z\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.180923 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qflzl\" (UniqueName: \"kubernetes.io/projected/da17354a-6849-474a-95c0-10c651c10d53-kube-api-access-qflzl\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.180923 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-image-registry-private-configuration\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.180923 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2k8\" (UniqueName: \"kubernetes.io/projected/7abba9dc-6fbe-4927-8298-ddda486d8ed1-kube-api-access-ss2k8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.180923 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4zn\" (UniqueName: \"kubernetes.io/projected/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-kube-api-access-ww4zn\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.180923 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180873 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-stats-auth\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.180923 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.180923 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180904 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcn6\" (UniqueName: \"kubernetes.io/projected/41109807-ec62-4222-ab2b-43a068961aab-kube-api-access-8tcn6\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-bound-sa-token\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41109807-ec62-4222-ab2b-43a068961aab-config\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr56\" (UniqueName: \"kubernetes.io/projected/77b46604-88b1-439a-b69d-4689aa67e1f8-kube-api-access-ztr56\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.180992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-config\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-trusted-ca\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qk7c\" (UniqueName: \"kubernetes.io/projected/6c5f9dc9-9bed-402b-af9f-16b918a054e5-kube-api-access-7qk7c\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181047 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5f9dc9-9bed-402b-af9f-16b918a054e5-tmp-dir\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9eea67d0-b1b9-4305-811e-20aa6ea233ed-snapshots\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:20.181133 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eea67d0-b1b9-4305-811e-20aa6ea233ed-tmp\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181166 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-certificates\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cc9c356-f4bd-409d-801a-d19df7559c6c-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7abba9dc-6fbe-4927-8298-ddda486d8ed1-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.181285 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181317 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eea67d0-b1b9-4305-811e-20aa6ea233ed-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.181387 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls podName:77b46604-88b1-439a-b69d-4689aa67e1f8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.681370499 +0000 UTC m=+34.493978842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-h422k" (UID: "77b46604-88b1-439a-b69d-4689aa67e1f8") : secret "samples-operator-tls" not found Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgsw8\" (UniqueName: \"kubernetes.io/projected/7cc9c356-f4bd-409d-801a-d19df7559c6c-kube-api-access-fgsw8\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-default-certificate\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.181585 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-serving-cert\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4srn\" (UniqueName: \"kubernetes.io/projected/4711a9be-591a-4346-a3ca-15156411555d-kube-api-access-v4srn\") pod \"volume-data-source-validator-695c895bd6-9sz65\" (UID: \"4711a9be-591a-4346-a3ca-15156411555d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eea67d0-b1b9-4305-811e-20aa6ea233ed-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e42973-63e2-40bd-ac95-43d33a321f99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p\" (UID: \"74e42973-63e2-40bd-ac95-43d33a321f99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181806 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxmb\" (UniqueName: \"kubernetes.io/projected/9eea67d0-b1b9-4305-811e-20aa6ea233ed-kube-api-access-6nxmb\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7abba9dc-6fbe-4927-8298-ddda486d8ed1-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181878 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-ca-trust-extracted\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.181901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-trusted-ca\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.182040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.182028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.182393 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.182059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-installation-pull-secrets\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.182393 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.182213 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:17:20.182393 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.182246 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9597f89b-ng6z7: secret "image-registry-tls" not found Sep 09 23:17:20.182393 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.182242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-ca-trust-extracted\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.182393 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.182338 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls podName:c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.682319034 +0000 UTC m=+34.494927390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls") pod "image-registry-7f9597f89b-ng6z7" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8") : secret "image-registry-tls" not found Sep 09 23:17:20.182812 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.182788 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-trusted-ca\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.185300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.185270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7abba9dc-6fbe-4927-8298-ddda486d8ed1-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.185378 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.185300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-installation-pull-secrets\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.185378 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.185302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-image-registry-private-configuration\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.190199 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.190175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-bound-sa-token\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.190631 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.190238 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztr56\" (UniqueName: \"kubernetes.io/projected/77b46604-88b1-439a-b69d-4689aa67e1f8-kube-api-access-ztr56\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:20.191251 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.191210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4srn\" (UniqueName: \"kubernetes.io/projected/4711a9be-591a-4346-a3ca-15156411555d-kube-api-access-v4srn\") pod \"volume-data-source-validator-695c895bd6-9sz65\" (UID: \"4711a9be-591a-4346-a3ca-15156411555d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" Sep 09 23:17:20.191344 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.191326 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-kube-api-access-2xd2z\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.191514 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.191492 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2k8\" (UniqueName: \"kubernetes.io/projected/7abba9dc-6fbe-4927-8298-ddda486d8ed1-kube-api-access-ss2k8\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hrjpm\" (UID: \"7abba9dc-6fbe-4927-8298-ddda486d8ed1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.206150 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206129 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f765574b6-292xd"] Sep 09 23:17:20.206239 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206160 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jpxs"] Sep 09 23:17:20.206239 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206178 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p"] Sep 09 23:17:20.206239 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206193 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-7ghfc"] Sep 09 23:17:20.206239 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206211 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg"] Sep 09 23:17:20.206239 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4x4rk"] Sep 09 23:17:20.206239 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206226 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.206239 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206236 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm"] Sep 09 23:17:20.206484 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206252 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn"] Sep 09 23:17:20.206484 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206264 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz"] Sep 09 23:17:20.206484 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.206289 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-v5mr2"] Sep 09 23:17:20.208673 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.208644 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 23:17:20.208776 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.208760 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 23:17:20.208836 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.208820 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wxhvm\"" Sep 09 23:17:20.208884 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.208825 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 23:17:20.220375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.220359 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-v5mr2"] Sep 09 23:17:20.220465 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.220440 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:20.222757 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.222738 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 09 23:17:20.222757 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.222748 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jdxq6\"" Sep 09 23:17:20.222884 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.222764 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 09 23:17:20.282881 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.282862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qflzl\" (UniqueName: \"kubernetes.io/projected/da17354a-6849-474a-95c0-10c651c10d53-kube-api-access-qflzl\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.282984 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.282890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6698e90f-06ad-4a2e-af70-ee5821fa0a0d-cert\") pod \"ingress-canary-4x4rk\" (UID: \"6698e90f-06ad-4a2e-af70-ee5821fa0a0d\") " pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.282984 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.282909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4zn\" (UniqueName: \"kubernetes.io/projected/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-kube-api-access-ww4zn\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.282984 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.282927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-stats-auth\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.282984 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.282970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.283187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcn6\" (UniqueName: \"kubernetes.io/projected/41109807-ec62-4222-ab2b-43a068961aab-kube-api-access-8tcn6\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.283187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283045 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hvw\" (UniqueName: \"kubernetes.io/projected/23398c6c-0e12-4d1e-992a-cd53788f7d86-kube-api-access-h2hvw\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.283187 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.283122 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:17:20.283187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zpd\" (UniqueName: \"kubernetes.io/projected/6698e90f-06ad-4a2e-af70-ee5821fa0a0d-kube-api-access-55zpd\") pod \"ingress-canary-4x4rk\" (UID: \"6698e90f-06ad-4a2e-af70-ee5821fa0a0d\") " pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.283181 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls podName:6c5f9dc9-9bed-402b-af9f-16b918a054e5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.783160809 +0000 UTC m=+34.595769160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls") pod "dns-default-8jpxs" (UID: "6c5f9dc9-9bed-402b-af9f-16b918a054e5") : secret "dns-default-metrics-tls" not found Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.283296 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.283348 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.783330235 +0000 UTC m=+34.595938586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : secret "router-metrics-certs-default" not found Sep 09 23:17:20.283375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283370 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23398c6c-0e12-4d1e-992a-cd53788f7d86-tmp\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41109807-ec62-4222-ab2b-43a068961aab-config\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283404 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98vck\" (UniqueName: \"kubernetes.io/projected/644c50af-e471-40d8-ab36-5195c8d8014c-kube-api-access-98vck\") pod \"network-check-source-685456655f-7ghfc\" (UID: \"644c50af-e471-40d8-ab36-5195c8d8014c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-config\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-trusted-ca\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qk7c\" (UniqueName: \"kubernetes.io/projected/6c5f9dc9-9bed-402b-af9f-16b918a054e5-kube-api-access-7qk7c\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-ca\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5f9dc9-9bed-402b-af9f-16b918a054e5-tmp-dir\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.283780 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9eea67d0-b1b9-4305-811e-20aa6ea233ed-snapshots\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eea67d0-b1b9-4305-811e-20aa6ea233ed-tmp\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cc9c356-f4bd-409d-801a-d19df7559c6c-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eea67d0-b1b9-4305-811e-20aa6ea233ed-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgsw8\" (UniqueName: \"kubernetes.io/projected/7cc9c356-f4bd-409d-801a-d19df7559c6c-kube-api-access-fgsw8\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283918 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41109807-ec62-4222-ab2b-43a068961aab-config\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/23398c6c-0e12-4d1e-992a-cd53788f7d86-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.283977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-default-certificate\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284052 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-serving-cert\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c5f9dc9-9bed-402b-af9f-16b918a054e5-tmp-dir\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eea67d0-b1b9-4305-811e-20aa6ea233ed-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e42973-63e2-40bd-ac95-43d33a321f99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p\" (UID: \"74e42973-63e2-40bd-ac95-43d33a321f99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.284162 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284163 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-config\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxmb\" (UniqueName: \"kubernetes.io/projected/9eea67d0-b1b9-4305-811e-20aa6ea233ed-kube-api-access-6nxmb\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eea67d0-b1b9-4305-811e-20aa6ea233ed-tmp\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-hub\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284271 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9eea67d0-b1b9-4305-811e-20aa6ea233ed-snapshots\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbtn\" (UniqueName: \"kubernetes.io/projected/74e42973-63e2-40bd-ac95-43d33a321f99-kube-api-access-5bbtn\") pod \"managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p\" (UID: \"74e42973-63e2-40bd-ac95-43d33a321f99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.284351 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.284406 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls podName:7cc9c356-f4bd-409d-801a-d19df7559c6c nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.784391464 +0000 UTC m=+34.596999810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-kh9wq" (UID: "7cc9c356-f4bd-409d-801a-d19df7559c6c") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.284423 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.784413833 +0000 UTC m=+34.597022184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : configmap references non-existent config key: service-ca.crt Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284451 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eea67d0-b1b9-4305-811e-20aa6ea233ed-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284624 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxst6\" (UniqueName: \"kubernetes.io/projected/67d1f8cf-5d76-4d43-b63c-97933be1a863-kube-api-access-xxst6\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9eea67d0-b1b9-4305-811e-20aa6ea233ed-serving-cert\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d1f8cf-5d76-4d43-b63c-97933be1a863-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/51a700c9-334f-4769-84f3-5e806b045aad-nginx-conf\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41109807-ec62-4222-ab2b-43a068961aab-serving-cert\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.284989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.284818 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f9dc9-9bed-402b-af9f-16b918a054e5-config-volume\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.285811 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.285177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7cc9c356-f4bd-409d-801a-d19df7559c6c-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.285811 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.285269 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-trusted-ca\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.285811 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.285749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9eea67d0-b1b9-4305-811e-20aa6ea233ed-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.286118 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.286094 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-stats-auth\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.286161 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.286104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c5f9dc9-9bed-402b-af9f-16b918a054e5-config-volume\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.287642 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.287583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-default-certificate\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.287857 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.287831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41109807-ec62-4222-ab2b-43a068961aab-serving-cert\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.288023 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.287990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/74e42973-63e2-40bd-ac95-43d33a321f99-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p\" (UID: \"74e42973-63e2-40bd-ac95-43d33a321f99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.288143 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.288022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9eea67d0-b1b9-4305-811e-20aa6ea233ed-serving-cert\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.288372 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.288350 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-serving-cert\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.292148 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.292101 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4zn\" (UniqueName: \"kubernetes.io/projected/9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5-kube-api-access-ww4zn\") pod \"console-operator-85b454874b-wq44d\" (UID: \"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5\") " pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.292706 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.292679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qflzl\" (UniqueName: \"kubernetes.io/projected/da17354a-6849-474a-95c0-10c651c10d53-kube-api-access-qflzl\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.292788 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.292701 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcn6\" (UniqueName: \"kubernetes.io/projected/41109807-ec62-4222-ab2b-43a068961aab-kube-api-access-8tcn6\") pod \"service-ca-operator-859f94c4b-j8vqn\" (UID: \"41109807-ec62-4222-ab2b-43a068961aab\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.293010 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.292988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxmb\" (UniqueName: \"kubernetes.io/projected/9eea67d0-b1b9-4305-811e-20aa6ea233ed-kube-api-access-6nxmb\") pod \"insights-operator-76bcd6ccf5-sktw2\" (UID: \"9eea67d0-b1b9-4305-811e-20aa6ea233ed\") " pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.293168 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.293143 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qk7c\" (UniqueName: \"kubernetes.io/projected/6c5f9dc9-9bed-402b-af9f-16b918a054e5-kube-api-access-7qk7c\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.293249 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.293145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgsw8\" (UniqueName: \"kubernetes.io/projected/7cc9c356-f4bd-409d-801a-d19df7559c6c-kube-api-access-fgsw8\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.293515 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.293497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbtn\" (UniqueName: \"kubernetes.io/projected/74e42973-63e2-40bd-ac95-43d33a321f99-kube-api-access-5bbtn\") pod \"managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p\" (UID: \"74e42973-63e2-40bd-ac95-43d33a321f99\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.335636 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.335557 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" Sep 09 23:17:20.342728 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.342709 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" Sep 09 23:17:20.350425 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.350410 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" Sep 09 23:17:20.366582 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.366550 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:20.386601 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.385355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxst6\" (UniqueName: \"kubernetes.io/projected/67d1f8cf-5d76-4d43-b63c-97933be1a863-kube-api-access-xxst6\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.386601 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:20.386601 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d1f8cf-5d76-4d43-b63c-97933be1a863-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.386805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/51a700c9-334f-4769-84f3-5e806b045aad-nginx-conf\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:20.386805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6698e90f-06ad-4a2e-af70-ee5821fa0a0d-cert\") pod \"ingress-canary-4x4rk\" (UID: \"6698e90f-06ad-4a2e-af70-ee5821fa0a0d\") " pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.386805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hvw\" (UniqueName: \"kubernetes.io/projected/23398c6c-0e12-4d1e-992a-cd53788f7d86-kube-api-access-h2hvw\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.386805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zpd\" (UniqueName: \"kubernetes.io/projected/6698e90f-06ad-4a2e-af70-ee5821fa0a0d-kube-api-access-55zpd\") pod \"ingress-canary-4x4rk\" (UID: \"6698e90f-06ad-4a2e-af70-ee5821fa0a0d\") " pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.386805 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.386745 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:17:20.386805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.386805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386793 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.387101 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.386830 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret podName:1a33b8ed-f089-4b62-b83c-23c8db52500b nodeName:}" failed. No retries permitted until 2026-09-09 23:17:52.38680971 +0000 UTC m=+66.199418066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret") pod "global-pull-secret-syncer-5rgz4" (UID: "1a33b8ed-f089-4b62-b83c-23c8db52500b") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:17:20.387101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:20.387101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23398c6c-0e12-4d1e-992a-cd53788f7d86-tmp\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.387101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98vck\" (UniqueName: \"kubernetes.io/projected/644c50af-e471-40d8-ab36-5195c8d8014c-kube-api-access-98vck\") pod \"network-check-source-685456655f-7ghfc\" (UID: \"644c50af-e471-40d8-ab36-5195c8d8014c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" Sep 09 23:17:20.387101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.386988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-ca\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.387101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.387060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/23398c6c-0e12-4d1e-992a-cd53788f7d86-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.387451 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.387105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:20.387451 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.387217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-hub\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.387564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.387540 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:20.388132 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.387861 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:17:20.388132 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.387925 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert podName:51a700c9-334f-4769-84f3-5e806b045aad nodeName:}" failed. No retries permitted until 2026-09-09 23:17:20.887906899 +0000 UTC m=+34.700515258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v5mr2" (UID: "51a700c9-334f-4769-84f3-5e806b045aad") : secret "networking-console-plugin-cert" not found Sep 09 23:17:20.388287 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.388234 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23398c6c-0e12-4d1e-992a-cd53788f7d86-tmp\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.388287 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.388277 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:17:20.388385 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.388346 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs podName:060a36f0-058c-4110-a094-dff084254fb5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:52.38831201 +0000 UTC m=+66.200920352 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs") pod "network-metrics-daemon-ht64b" (UID: "060a36f0-058c-4110-a094-dff084254fb5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:17:20.389103 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.388882 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d1f8cf-5d76-4d43-b63c-97933be1a863-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.389103 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.389091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/51a700c9-334f-4769-84f3-5e806b045aad-nginx-conf\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:20.390734 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.390709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/23398c6c-0e12-4d1e-992a-cd53788f7d86-klusterlet-config\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.390878 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.390861 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6698e90f-06ad-4a2e-af70-ee5821fa0a0d-cert\") pod \"ingress-canary-4x4rk\" (UID: \"6698e90f-06ad-4a2e-af70-ee5821fa0a0d\") " pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.391385 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.391364 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.391774 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.391730 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-ca\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.391939 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.391922 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-hub\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.391993 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.391933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/67d1f8cf-5d76-4d43-b63c-97933be1a863-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.395176 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.395156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxst6\" (UniqueName: \"kubernetes.io/projected/67d1f8cf-5d76-4d43-b63c-97933be1a863-kube-api-access-xxst6\") pod \"cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz\" (UID: \"67d1f8cf-5d76-4d43-b63c-97933be1a863\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.395258 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.395181 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zpd\" (UniqueName: \"kubernetes.io/projected/6698e90f-06ad-4a2e-af70-ee5821fa0a0d-kube-api-access-55zpd\") pod \"ingress-canary-4x4rk\" (UID: \"6698e90f-06ad-4a2e-af70-ee5821fa0a0d\") " pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.395983 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.395962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98vck\" (UniqueName: \"kubernetes.io/projected/644c50af-e471-40d8-ab36-5195c8d8014c-kube-api-access-98vck\") pod \"network-check-source-685456655f-7ghfc\" (UID: \"644c50af-e471-40d8-ab36-5195c8d8014c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" Sep 09 23:17:20.398559 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.398538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hvw\" (UniqueName: \"kubernetes.io/projected/23398c6c-0e12-4d1e-992a-cd53788f7d86-kube-api-access-h2hvw\") pod \"klusterlet-addon-workmgr-7bc56b8b8f-s9pwg\" (UID: \"23398c6c-0e12-4d1e-992a-cd53788f7d86\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.416545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.416527 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" Sep 09 23:17:20.429077 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.429020 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" Sep 09 23:17:20.449424 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.449404 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" Sep 09 23:17:20.451890 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.451858 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:20.465732 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.465718 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:20.483952 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.483934 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:17:20.488323 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.488308 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:20.491021 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.491001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsjr\" (UniqueName: \"kubernetes.io/projected/81c0e958-8a2c-47cc-804c-50e383fe3b03-kube-api-access-9bsjr\") pod \"network-check-target-bdjlj\" (UID: \"81c0e958-8a2c-47cc-804c-50e383fe3b03\") " pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:20.528101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.528081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4x4rk" Sep 09 23:17:20.641527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.637949 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:20.641527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.638644 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:20.641527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.639319 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:20.641527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.640975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:17:20.642259 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.642090 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-snlxf\"" Sep 09 23:17:20.642259 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.642106 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n28fd\"" Sep 09 23:17:20.642427 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.642335 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:17:20.662260 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.662180 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65"] Sep 09 23:17:20.663808 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.663784 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm"] Sep 09 23:17:20.682522 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.682348 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:20.693638 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.692922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:20.693638 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.693055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:20.693638 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.693219 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 23:17:20.693638 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.693272 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls podName:77b46604-88b1-439a-b69d-4689aa67e1f8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:21.693254103 +0000 UTC m=+35.505862458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-h422k" (UID: "77b46604-88b1-439a-b69d-4689aa67e1f8") : secret "samples-operator-tls" not found Sep 09 23:17:20.693983 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.693676 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:17:20.693983 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.693690 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9597f89b-ng6z7: secret "image-registry-tls" not found Sep 09 23:17:20.693983 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.693731 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls podName:c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:21.693717048 +0000 UTC m=+35.506325392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls") pod "image-registry-7f9597f89b-ng6z7" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8") : secret "image-registry-tls" not found Sep 09 23:17:20.734241 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:20.734204 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abba9dc_6fbe_4927_8298_ddda486d8ed1.slice/crio-2e3f872a6d88f34d934492787fb66046a06f16a5c35da6215226319d60ef5eae WatchSource:0}: Error finding container 2e3f872a6d88f34d934492787fb66046a06f16a5c35da6215226319d60ef5eae: Status 404 returned error can't find the container with id 2e3f872a6d88f34d934492787fb66046a06f16a5c35da6215226319d60ef5eae Sep 09 23:17:20.756683 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.756558 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:20.792844 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.792761 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" event={"ID":"7abba9dc-6fbe-4927-8298-ddda486d8ed1","Type":"ContainerStarted","Data":"2e3f872a6d88f34d934492787fb66046a06f16a5c35da6215226319d60ef5eae"} Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.793672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.793723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.793827 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.793847 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.793881 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls podName:6c5f9dc9-9bed-402b-af9f-16b918a054e5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:21.793864132 +0000 UTC m=+35.606472472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls") pod "dns-default-8jpxs" (UID: "6c5f9dc9-9bed-402b-af9f-16b918a054e5") : secret "dns-default-metrics-tls" not found Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.793938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.793966 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:21.793946962 +0000 UTC m=+35.606555314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : secret "router-metrics-certs-default" not found Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.794010 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:21.793999192 +0000 UTC m=+35.606607557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : configmap references non-existent config key: service-ca.crt Sep 09 23:17:20.794194 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.794002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:20.794759 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.794395 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:20.794759 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.794451 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls podName:7cc9c356-f4bd-409d-801a-d19df7559c6c nodeName:}" failed. No retries permitted until 2026-09-09 23:17:21.794434891 +0000 UTC m=+35.607043245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-kh9wq" (UID: "7cc9c356-f4bd-409d-801a-d19df7559c6c") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:20.795251 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.795230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" event={"ID":"4711a9be-591a-4346-a3ca-15156411555d","Type":"ContainerStarted","Data":"ea2f252bc1c11e1f369425d31d1b21f08e7c13fee28939817f0bef26235e031c"} Sep 09 23:17:20.844049 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.844016 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-sktw2"] Sep 09 23:17:20.846950 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.846926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:20.877902 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.877876 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn"] Sep 09 23:17:20.890941 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.890916 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-7ghfc"] Sep 09 23:17:20.894095 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.894022 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p"] Sep 09 23:17:20.895532 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.895510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:20.895704 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.895685 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:17:20.895776 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:20.895763 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert podName:51a700c9-334f-4769-84f3-5e806b045aad nodeName:}" failed. No retries permitted until 2026-09-09 23:17:21.89574176 +0000 UTC m=+35.708350118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v5mr2" (UID: "51a700c9-334f-4769-84f3-5e806b045aad") : secret "networking-console-plugin-cert" not found Sep 09 23:17:20.914584 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.914565 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4x4rk"] Sep 09 23:17:20.916887 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.916869 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-wq44d"] Sep 09 23:17:20.949638 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.949619 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg"] Sep 09 23:17:20.952636 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:20.952616 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz"] Sep 09 23:17:21.001416 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.001394 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bdjlj"] Sep 09 23:17:21.004548 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.004524 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a26c0c0_6ec6_4b20_8bb7_b1dcc9fa27e5.slice/crio-af354fa0888b0c854c033f706d66a3a2bd0eb35c1145950401148e89b3c9bd4f WatchSource:0}: Error finding container af354fa0888b0c854c033f706d66a3a2bd0eb35c1145950401148e89b3c9bd4f: Status 404 returned error can't find the container with id af354fa0888b0c854c033f706d66a3a2bd0eb35c1145950401148e89b3c9bd4f Sep 09 23:17:21.005273 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.005156 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67d1f8cf_5d76_4d43_b63c_97933be1a863.slice/crio-b7faeb5d6d8e71c3112c63c9047173e05ccd6ef3e7eb41b5536d1e18866cf3eb WatchSource:0}: Error finding container b7faeb5d6d8e71c3112c63c9047173e05ccd6ef3e7eb41b5536d1e18866cf3eb: Status 404 returned error can't find the container with id b7faeb5d6d8e71c3112c63c9047173e05ccd6ef3e7eb41b5536d1e18866cf3eb Sep 09 23:17:21.005899 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.005874 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c0e958_8a2c_47cc_804c_50e383fe3b03.slice/crio-c41c50a997cb9f92d66c546f2403e421a272adf524c3f908e546ba75c0d80507 WatchSource:0}: Error finding container c41c50a997cb9f92d66c546f2403e421a272adf524c3f908e546ba75c0d80507: Status 404 returned error can't find the container with id c41c50a997cb9f92d66c546f2403e421a272adf524c3f908e546ba75c0d80507 Sep 09 23:17:21.006992 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.006940 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eea67d0_b1b9_4305_811e_20aa6ea233ed.slice/crio-caef207f879a475ebff8348e82ae6aaf1653d98eb074a564c114c2c5d1bc50c3 WatchSource:0}: Error finding container caef207f879a475ebff8348e82ae6aaf1653d98eb074a564c114c2c5d1bc50c3: Status 404 returned error can't find the container with id caef207f879a475ebff8348e82ae6aaf1653d98eb074a564c114c2c5d1bc50c3 Sep 09 23:17:21.007758 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.007731 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41109807_ec62_4222_ab2b_43a068961aab.slice/crio-3e35b14b7f866f5360866e3bc40d00fafe9bd185798ef1f05de4775232022864 WatchSource:0}: Error finding container 3e35b14b7f866f5360866e3bc40d00fafe9bd185798ef1f05de4775232022864: Status 404 returned error can't find the container with id 3e35b14b7f866f5360866e3bc40d00fafe9bd185798ef1f05de4775232022864 Sep 09 23:17:21.009858 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.009809 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23398c6c_0e12_4d1e_992a_cd53788f7d86.slice/crio-998a071d721b6f33bb8a3ccd56f648ba5e193c52af481bce4198ba4cbfd281a9 WatchSource:0}: Error finding container 998a071d721b6f33bb8a3ccd56f648ba5e193c52af481bce4198ba4cbfd281a9: Status 404 returned error can't find the container with id 998a071d721b6f33bb8a3ccd56f648ba5e193c52af481bce4198ba4cbfd281a9 Sep 09 23:17:21.012274 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.012003 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74e42973_63e2_40bd_ac95_43d33a321f99.slice/crio-2c16ec4f3e2afc32c0f8453c0ca357df6a593927fad4c3bd62588810f0693324 WatchSource:0}: Error finding container 2c16ec4f3e2afc32c0f8453c0ca357df6a593927fad4c3bd62588810f0693324: Status 404 returned error can't find the container with id 2c16ec4f3e2afc32c0f8453c0ca357df6a593927fad4c3bd62588810f0693324 Sep 09 23:17:21.014106 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:21.013740 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644c50af_e471_40d8_ab36_5195c8d8014c.slice/crio-d93d442094cd09c79497f48d9cfac52a2c6ad4e3b6c219a859a3cc7d2a918e2a WatchSource:0}: Error finding container d93d442094cd09c79497f48d9cfac52a2c6ad4e3b6c219a859a3cc7d2a918e2a: Status 404 returned error can't find the container with id d93d442094cd09c79497f48d9cfac52a2c6ad4e3b6c219a859a3cc7d2a918e2a Sep 09 23:17:21.581744 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.581706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:21.704149 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.703877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:21.704149 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.703991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:21.704149 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.704111 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 23:17:21.704427 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.704172 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls podName:77b46604-88b1-439a-b69d-4689aa67e1f8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:23.704157552 +0000 UTC m=+37.516765910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-h422k" (UID: "77b46604-88b1-439a-b69d-4689aa67e1f8") : secret "samples-operator-tls" not found Sep 09 23:17:21.704427 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.704187 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:17:21.704427 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.704205 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9597f89b-ng6z7: secret "image-registry-tls" not found Sep 09 23:17:21.704427 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.704255 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls podName:c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:23.704238286 +0000 UTC m=+37.516846640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls") pod "image-registry-7f9597f89b-ng6z7" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8") : secret "image-registry-tls" not found Sep 09 23:17:21.804581 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.804547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.804764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.804774 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:23.804749869 +0000 UTC m=+37.617358212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : configmap references non-existent config key: service-ca.crt Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.804910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.804950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.805107 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.805159 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:23.80514469 +0000 UTC m=+37.617753045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : secret "router-metrics-certs-default" not found Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.805184 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.805220 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.805239 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls podName:7cc9c356-f4bd-409d-801a-d19df7559c6c nodeName:}" failed. No retries permitted until 2026-09-09 23:17:23.805223683 +0000 UTC m=+37.617832043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-kh9wq" (UID: "7cc9c356-f4bd-409d-801a-d19df7559c6c") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:21.805293 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.805259 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls podName:6c5f9dc9-9bed-402b-af9f-16b918a054e5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:23.805246498 +0000 UTC m=+37.617854858 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls") pod "dns-default-8jpxs" (UID: "6c5f9dc9-9bed-402b-af9f-16b918a054e5") : secret "dns-default-metrics-tls" not found Sep 09 23:17:21.806274 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.806105 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wq44d" event={"ID":"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5","Type":"ContainerStarted","Data":"af354fa0888b0c854c033f706d66a3a2bd0eb35c1145950401148e89b3c9bd4f"} Sep 09 23:17:21.811207 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.810322 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4x4rk" event={"ID":"6698e90f-06ad-4a2e-af70-ee5821fa0a0d","Type":"ContainerStarted","Data":"80bcf06317a79c87a286b0b746124a7e89837de8a9e9db3a95cb8ccadf675da1"} Sep 09 23:17:21.818372 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.817015 2559 generic.go:356] "Generic (PLEG): container finished" podID="14ee243b-7559-462e-9414-a47ed4df6877" containerID="032837d96b2cb21349745c97beb13d663ff8911aa7316786b4ca7f69378aa444" exitCode=0 Sep 09 23:17:21.818372 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.817090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerDied","Data":"032837d96b2cb21349745c97beb13d663ff8911aa7316786b4ca7f69378aa444"} Sep 09 23:17:21.822377 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.822281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" event={"ID":"74e42973-63e2-40bd-ac95-43d33a321f99","Type":"ContainerStarted","Data":"2c16ec4f3e2afc32c0f8453c0ca357df6a593927fad4c3bd62588810f0693324"} Sep 09 23:17:21.824770 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.824746 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" event={"ID":"67d1f8cf-5d76-4d43-b63c-97933be1a863","Type":"ContainerStarted","Data":"b7faeb5d6d8e71c3112c63c9047173e05ccd6ef3e7eb41b5536d1e18866cf3eb"} Sep 09 23:17:21.826871 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.826853 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" event={"ID":"9eea67d0-b1b9-4305-811e-20aa6ea233ed","Type":"ContainerStarted","Data":"caef207f879a475ebff8348e82ae6aaf1653d98eb074a564c114c2c5d1bc50c3"} Sep 09 23:17:21.831124 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.829894 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" event={"ID":"23398c6c-0e12-4d1e-992a-cd53788f7d86","Type":"ContainerStarted","Data":"998a071d721b6f33bb8a3ccd56f648ba5e193c52af481bce4198ba4cbfd281a9"} Sep 09 23:17:21.832929 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.832858 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" event={"ID":"644c50af-e471-40d8-ab36-5195c8d8014c","Type":"ContainerStarted","Data":"d93d442094cd09c79497f48d9cfac52a2c6ad4e3b6c219a859a3cc7d2a918e2a"} Sep 09 23:17:21.838285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.838219 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdjlj" event={"ID":"81c0e958-8a2c-47cc-804c-50e383fe3b03","Type":"ContainerStarted","Data":"c41c50a997cb9f92d66c546f2403e421a272adf524c3f908e546ba75c0d80507"} Sep 09 23:17:21.841720 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.841693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" event={"ID":"41109807-ec62-4222-ab2b-43a068961aab","Type":"ContainerStarted","Data":"3e35b14b7f866f5360866e3bc40d00fafe9bd185798ef1f05de4775232022864"} Sep 09 23:17:21.906439 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.906334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:21.906878 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.906782 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:17:21.906878 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:21.906845 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert podName:51a700c9-334f-4769-84f3-5e806b045aad nodeName:}" failed. No retries permitted until 2026-09-09 23:17:23.906825218 +0000 UTC m=+37.719433572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v5mr2" (UID: "51a700c9-334f-4769-84f3-5e806b045aad") : secret "networking-console-plugin-cert" not found Sep 09 23:17:21.943177 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:21.943123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:22.044759 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:22.044603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:23.736457 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:23.736197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:23.736947 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:23.736624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:23.736947 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.736825 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 23:17:23.736947 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.736894 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls podName:77b46604-88b1-439a-b69d-4689aa67e1f8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:27.736874054 +0000 UTC m=+41.549482408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-h422k" (UID: "77b46604-88b1-439a-b69d-4689aa67e1f8") : secret "samples-operator-tls" not found Sep 09 23:17:23.737375 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.737357 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:17:23.737445 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.737379 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9597f89b-ng6z7: secret "image-registry-tls" not found Sep 09 23:17:23.737445 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.737427 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls podName:c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:27.737410991 +0000 UTC m=+41.550019333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls") pod "image-registry-7f9597f89b-ng6z7" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8") : secret "image-registry-tls" not found Sep 09 23:17:23.837132 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:23.837100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:23.837261 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:23.837157 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:23.837321 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.837295 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:17:23.837321 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.837305 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:17:23.837408 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.837351 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:27.837333916 +0000 UTC m=+41.649942276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : secret "router-metrics-certs-default" not found Sep 09 23:17:23.837408 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:23.837384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:23.837533 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:23.837411 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:23.837533 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.837427 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls podName:6c5f9dc9-9bed-402b-af9f-16b918a054e5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:27.837406203 +0000 UTC m=+41.650014558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls") pod "dns-default-8jpxs" (UID: "6c5f9dc9-9bed-402b-af9f-16b918a054e5") : secret "dns-default-metrics-tls" not found Sep 09 23:17:23.837533 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.837475 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:23.837533 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.837508 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls podName:7cc9c356-f4bd-409d-801a-d19df7559c6c nodeName:}" failed. No retries permitted until 2026-09-09 23:17:27.837498204 +0000 UTC m=+41.650106546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-kh9wq" (UID: "7cc9c356-f4bd-409d-801a-d19df7559c6c") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:23.837761 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.837611 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:27.83758575 +0000 UTC m=+41.650194092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : configmap references non-existent config key: service-ca.crt Sep 09 23:17:23.940231 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:23.939026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:23.940231 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.939242 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:17:23.940231 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:23.939308 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert podName:51a700c9-334f-4769-84f3-5e806b045aad nodeName:}" failed. No retries permitted until 2026-09-09 23:17:27.939288952 +0000 UTC m=+41.751897295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v5mr2" (UID: "51a700c9-334f-4769-84f3-5e806b045aad") : secret "networking-console-plugin-cert" not found Sep 09 23:17:25.293749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:25.293715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:25.354943 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:25.354908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:26.401894 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:26.401786 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:26.456486 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:26.456447 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:27.780213 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:27.780169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:27.780612 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:27.780278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:27.780612 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.780307 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 23:17:27.780612 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.780381 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls podName:77b46604-88b1-439a-b69d-4689aa67e1f8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.780359741 +0000 UTC m=+49.592968100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-h422k" (UID: "77b46604-88b1-439a-b69d-4689aa67e1f8") : secret "samples-operator-tls" not found Sep 09 23:17:27.780612 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.780421 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:17:27.780612 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.780437 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9597f89b-ng6z7: secret "image-registry-tls" not found Sep 09 23:17:27.780612 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.780490 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls podName:c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.780475325 +0000 UTC m=+49.593083668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls") pod "image-registry-7f9597f89b-ng6z7" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8") : secret "image-registry-tls" not found Sep 09 23:17:27.881110 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:27.881082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:27.881110 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:27.881110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:27.881274 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:27.881168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:27.881274 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:27.881193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:27.881274 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.881238 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.881221328 +0000 UTC m=+49.693829673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : configmap references non-existent config key: service-ca.crt Sep 09 23:17:27.881382 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.881289 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:27.881382 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.881297 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:17:27.881382 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.881322 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:17:27.881382 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.881332 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls podName:7cc9c356-f4bd-409d-801a-d19df7559c6c nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.881321174 +0000 UTC m=+49.693929514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-kh9wq" (UID: "7cc9c356-f4bd-409d-801a-d19df7559c6c") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:27.881382 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.881345 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.881339521 +0000 UTC m=+49.693947861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : secret "router-metrics-certs-default" not found Sep 09 23:17:27.881382 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.881369 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls podName:6c5f9dc9-9bed-402b-af9f-16b918a054e5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.881356943 +0000 UTC m=+49.693965287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls") pod "dns-default-8jpxs" (UID: "6c5f9dc9-9bed-402b-af9f-16b918a054e5") : secret "dns-default-metrics-tls" not found Sep 09 23:17:27.982278 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:27.982248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:27.982392 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.982357 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:17:27.982444 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:27.982412 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert podName:51a700c9-334f-4769-84f3-5e806b045aad nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.982400741 +0000 UTC m=+49.795009081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v5mr2" (UID: "51a700c9-334f-4769-84f3-5e806b045aad") : secret "networking-console-plugin-cert" not found Sep 09 23:17:30.583467 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:30.583385 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:30.684309 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:30.684259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:32.879667 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.879628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" event={"ID":"644c50af-e471-40d8-ab36-5195c8d8014c","Type":"ContainerStarted","Data":"adff7f095ec56bf7d9f3766c0926b3511428bd183841a4116d1a8ec165d1439a"} Sep 09 23:17:32.881641 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.881615 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" event={"ID":"7abba9dc-6fbe-4927-8298-ddda486d8ed1","Type":"ContainerStarted","Data":"4d364ea2a709e6abcf66a920629603f5675c6f47357ad8e6031ed1e2fd284282"} Sep 09 23:17:32.883418 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.883392 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bdjlj" event={"ID":"81c0e958-8a2c-47cc-804c-50e383fe3b03","Type":"ContainerStarted","Data":"ebe8bf40c4ca9e40c6d198ade41778a66e7db9ae878dfe16231d0aad7220f940"} Sep 09 23:17:32.883487 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.883447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:17:32.885361 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.885315 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" event={"ID":"41109807-ec62-4222-ab2b-43a068961aab","Type":"ContainerStarted","Data":"d50cda0e0aa1b2979eee28cb9567d01b28d63789eb2641ea2a0250296b14701a"} Sep 09 23:17:32.887519 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.887500 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/0.log" Sep 09 23:17:32.887624 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.887537 2559 generic.go:356] "Generic (PLEG): container finished" podID="9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5" containerID="8e9dcfdf77306a069d0ab2518b4ee0b8f284b1660f770104e5763fe28865b3f9" exitCode=255 Sep 09 23:17:32.887845 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.887642 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wq44d" event={"ID":"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5","Type":"ContainerDied","Data":"8e9dcfdf77306a069d0ab2518b4ee0b8f284b1660f770104e5763fe28865b3f9"} Sep 09 23:17:32.887919 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.887847 2559 scope.go:117] "RemoveContainer" containerID="8e9dcfdf77306a069d0ab2518b4ee0b8f284b1660f770104e5763fe28865b3f9" Sep 09 23:17:32.889703 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.889664 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4x4rk" event={"ID":"6698e90f-06ad-4a2e-af70-ee5821fa0a0d","Type":"ContainerStarted","Data":"95f6838a25509a9ff46c757ee4d15a11405158dc31dea22d514da769467184cf"} Sep 09 23:17:32.897085 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.897004 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-7ghfc" podStartSLOduration=29.643730351 podStartE2EDuration="40.896988827s" totalImagesPullingTime="11.253258476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:32.896772477 +0000 UTC m=+46.709380840" watchObservedRunningTime="2026-09-09 23:17:32.896988827 +0000 UTC m=+46.709597190" Sep 09 23:17:32.899452 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.899394 2559 generic.go:356] "Generic (PLEG): container finished" podID="14ee243b-7559-462e-9414-a47ed4df6877" containerID="b376b75a04e1a8c095e35604cfc82f8d3b534c55b4f0b2a785ba8ca8d15d8bcc" exitCode=0 Sep 09 23:17:32.899607 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.899473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerDied","Data":"b376b75a04e1a8c095e35604cfc82f8d3b534c55b4f0b2a785ba8ca8d15d8bcc"} Sep 09 23:17:32.902446 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.902338 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" event={"ID":"4711a9be-591a-4346-a3ca-15156411555d","Type":"ContainerStarted","Data":"f35df24f116e9a7332f4032c377a972dcec10728448020787ea88ce05ef10d51"} Sep 09 23:17:32.904051 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.904022 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" event={"ID":"74e42973-63e2-40bd-ac95-43d33a321f99","Type":"ContainerStarted","Data":"586e4634caac4eb0c12c2bf6c6f55caff59309990d6cea979da47f2bfa1edcb1"} Sep 09 23:17:32.906334 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.906312 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" event={"ID":"67d1f8cf-5d76-4d43-b63c-97933be1a863","Type":"ContainerStarted","Data":"0d90e9a66ec4ae26bac6a8b75ef0ced740f15437b7e3dfbafc5452f44ec06661"} Sep 09 23:17:32.906434 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.906345 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" event={"ID":"67d1f8cf-5d76-4d43-b63c-97933be1a863","Type":"ContainerStarted","Data":"d0bd668d2c1c2896de93877a4137517413e5f31ae08ac4e0be9035b016813615"} Sep 09 23:17:32.906434 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.906362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" event={"ID":"67d1f8cf-5d76-4d43-b63c-97933be1a863","Type":"ContainerStarted","Data":"a3ff0263381dcc1ff1d9d891fdd02e5c7a525cd6e35eb996b641701ad487445e"} Sep 09 23:17:32.907808 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.907787 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" event={"ID":"9eea67d0-b1b9-4305-811e-20aa6ea233ed","Type":"ContainerStarted","Data":"b4f8ac461fad97466ce55f1a39f99206d2dbd227e1db133dab1773c5eb220530"} Sep 09 23:17:32.928848 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.928735 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bdjlj" podStartSLOduration=34.676401334 podStartE2EDuration="45.928717303s" totalImagesPullingTime="11.252315969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:32.926888883 +0000 UTC m=+46.739497243" watchObservedRunningTime="2026-09-09 23:17:32.928717303 +0000 UTC m=+46.741325663" Sep 09 23:17:32.942478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.941902 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" podStartSLOduration=29.691169712 podStartE2EDuration="40.94189016s" totalImagesPullingTime="11.250720448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:32.941052036 +0000 UTC m=+46.753660398" watchObservedRunningTime="2026-09-09 23:17:32.94189016 +0000 UTC m=+46.754498523" Sep 09 23:17:32.981946 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:32.981876 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4x4rk" podStartSLOduration=2.753781707 podStartE2EDuration="13.981858083s" totalImagesPullingTime="11.228076376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:32.963315934 +0000 UTC m=+46.775924295" watchObservedRunningTime="2026-09-09 23:17:32.981858083 +0000 UTC m=+46.794466445" Sep 09 23:17:33.009645 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.009289 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9sz65" podStartSLOduration=32.106589932 podStartE2EDuration="42.009270692s" totalImagesPullingTime="9.90268076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:33.008779379 +0000 UTC m=+46.821387744" watchObservedRunningTime="2026-09-09 23:17:33.009270692 +0000 UTC m=+46.821879055" Sep 09 23:17:33.009974 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.009931 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" podStartSLOduration=30.480023369 podStartE2EDuration="42.009920834s" totalImagesPullingTime="11.529897465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:32.985184892 +0000 UTC m=+46.797793255" watchObservedRunningTime="2026-09-09 23:17:33.009920834 +0000 UTC m=+46.822529196" Sep 09 23:17:33.026514 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.026470 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" podStartSLOduration=10.75338699 podStartE2EDuration="22.026459951s" totalImagesPullingTime="11.273072961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:33.025323873 +0000 UTC m=+46.837932236" watchObservedRunningTime="2026-09-09 23:17:33.026459951 +0000 UTC m=+46.839068316" Sep 09 23:17:33.043852 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.043809 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cdc87fdfd-wjr2p" podStartSLOduration=10.791236929 podStartE2EDuration="22.043795732s" totalImagesPullingTime="11.252558803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:33.042836371 +0000 UTC m=+46.855444735" watchObservedRunningTime="2026-09-09 23:17:33.043795732 +0000 UTC m=+46.856404095" Sep 09 23:17:33.088711 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.088633 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" podStartSLOduration=30.899341902 podStartE2EDuration="42.088584406s" totalImagesPullingTime="11.189242504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:33.088125494 +0000 UTC m=+46.900733855" watchObservedRunningTime="2026-09-09 23:17:33.088584406 +0000 UTC m=+46.901192768" Sep 09 23:17:33.101798 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.101730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:33.206655 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.206622 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:33.565534 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.565465 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pf8kf"] Sep 09 23:17:33.587606 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.587569 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pf8kf"] Sep 09 23:17:33.587746 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.587691 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.590160 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.590117 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 23:17:33.590300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.590189 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 23:17:33.590300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.590191 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnpcr\"" Sep 09 23:17:33.612183 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.612158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:33.634481 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.634458 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42127793-921c-4220-b533-1797ac467275-crio-socket\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.634582 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.634501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45kt\" (UniqueName: \"kubernetes.io/projected/42127793-921c-4220-b533-1797ac467275-kube-api-access-x45kt\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.634647 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.634606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42127793-921c-4220-b533-1797ac467275-data-volume\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.634762 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.634746 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.634798 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.634778 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.665855 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.665822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:33.735537 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.735511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42127793-921c-4220-b533-1797ac467275-data-volume\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.735666 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.735629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.735666 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.735657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.735783 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.735754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42127793-921c-4220-b533-1797ac467275-crio-socket\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.735783 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:33.735764 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:17:33.735884 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.735786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45kt\" (UniqueName: \"kubernetes.io/projected/42127793-921c-4220-b533-1797ac467275-kube-api-access-x45kt\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.735884 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:33.735828 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:34.235809087 +0000 UTC m=+48.048417440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : configmap "kube-rbac-proxy" not found Sep 09 23:17:33.735884 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.735877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42127793-921c-4220-b533-1797ac467275-data-volume\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.736040 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.735901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42127793-921c-4220-b533-1797ac467275-crio-socket\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.736040 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:33.735983 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:17:33.736141 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:33.736040 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:34.236024148 +0000 UTC m=+48.048632494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : secret "insights-runtime-extractor-tls" not found Sep 09 23:17:33.747004 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.746981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45kt\" (UniqueName: \"kubernetes.io/projected/42127793-921c-4220-b533-1797ac467275-kube-api-access-x45kt\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:33.916732 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.916664 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:17:33.917191 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.917172 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/0.log" Sep 09 23:17:33.917236 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.917217 2559 generic.go:356] "Generic (PLEG): container finished" podID="9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5" containerID="130fc6af67d07d12b03ef35b3ce7afd9e8e72b82ea2b66bacb3233010b5f38d7" exitCode=255 Sep 09 23:17:33.917322 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.917302 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wq44d" event={"ID":"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5","Type":"ContainerDied","Data":"130fc6af67d07d12b03ef35b3ce7afd9e8e72b82ea2b66bacb3233010b5f38d7"} Sep 09 23:17:33.917379 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.917365 2559 scope.go:117] "RemoveContainer" containerID="8e9dcfdf77306a069d0ab2518b4ee0b8f284b1660f770104e5763fe28865b3f9" Sep 09 23:17:33.917583 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.917565 2559 scope.go:117] "RemoveContainer" containerID="130fc6af67d07d12b03ef35b3ce7afd9e8e72b82ea2b66bacb3233010b5f38d7" Sep 09 23:17:33.917818 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:33.917795 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-wq44d_openshift-console-operator(9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5)\"" pod="openshift-console-operator/console-operator-85b454874b-wq44d" podUID="9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5" Sep 09 23:17:33.921353 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.921314 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28zfb" event={"ID":"14ee243b-7559-462e-9414-a47ed4df6877","Type":"ContainerStarted","Data":"d026fe1a618c50a8434fa1c96343a1301930c2675890ee64894c000ff9a41d42"} Sep 09 23:17:33.954375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.954328 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-28zfb" podStartSLOduration=17.792229061 podStartE2EDuration="46.95430791s" totalImagesPullingTime="29.162078849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:33.95246222 +0000 UTC m=+47.765070585" watchObservedRunningTime="2026-09-09 23:17:33.95430791 +0000 UTC m=+47.766916274" Sep 09 23:17:33.956556 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:33.956532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.076960 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.076934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.177212 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.177186 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.241335 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.241309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:34.241335 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.241339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:34.241507 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:34.241440 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:17:34.241507 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:34.241501 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.241485406 +0000 UTC m=+49.054093745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : configmap "kube-rbac-proxy" not found Sep 09 23:17:34.241578 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:34.241517 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:17:34.241578 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:34.241557 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:35.241546437 +0000 UTC m=+49.054154777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : secret "insights-runtime-extractor-tls" not found Sep 09 23:17:34.365497 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.365455 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.400982 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.400952 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b"] Sep 09 23:17:34.436042 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.435974 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b"] Sep 09 23:17:34.436173 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.436089 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" Sep 09 23:17:34.439017 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.438987 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 09 23:17:34.439017 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.439021 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 09 23:17:34.439448 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.439432 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2jxxz\"" Sep 09 23:17:34.454563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.454540 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.513603 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.513561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.543958 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.543923 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjhj\" (UniqueName: \"kubernetes.io/projected/af5bcce7-094c-4f73-a233-d25bb0ecfb73-kube-api-access-bgjhj\") pod \"migrator-fdbb8fdff-kd55b\" (UID: \"af5bcce7-094c-4f73-a233-d25bb0ecfb73\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" Sep 09 23:17:34.552238 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.552219 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b2z6r_633f88c9-63c9-48d4-9ffc-523090d75649/dns-node-resolver/0.log" Sep 09 23:17:34.645022 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.644987 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjhj\" (UniqueName: \"kubernetes.io/projected/af5bcce7-094c-4f73-a233-d25bb0ecfb73-kube-api-access-bgjhj\") pod \"migrator-fdbb8fdff-kd55b\" (UID: \"af5bcce7-094c-4f73-a233-d25bb0ecfb73\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" Sep 09 23:17:34.661310 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.661275 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.661631 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.661584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjhj\" (UniqueName: \"kubernetes.io/projected/af5bcce7-094c-4f73-a233-d25bb0ecfb73-kube-api-access-bgjhj\") pod \"migrator-fdbb8fdff-kd55b\" (UID: \"af5bcce7-094c-4f73-a233-d25bb0ecfb73\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" Sep 09 23:17:34.745381 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.745320 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" Sep 09 23:17:34.764025 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.763996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" limit=1024 Sep 09 23:17:34.926424 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.926393 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:17:34.926876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:34.926824 2559 scope.go:117] "RemoveContainer" containerID="130fc6af67d07d12b03ef35b3ce7afd9e8e72b82ea2b66bacb3233010b5f38d7" Sep 09 23:17:34.927041 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:34.927015 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-wq44d_openshift-console-operator(9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5)\"" pod="openshift-console-operator/console-operator-85b454874b-wq44d" podUID="9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5" Sep 09 23:17:35.035112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.035087 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b"] Sep 09 23:17:35.251951 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.251921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:35.251951 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.251953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:35.252140 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.252060 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:17:35.252140 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.252107 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:37.252093371 +0000 UTC m=+51.064701715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : secret "insights-runtime-extractor-tls" not found Sep 09 23:17:35.252140 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.252110 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:17:35.252237 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.252153 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:37.252142002 +0000 UTC m=+51.064750348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : configmap "kube-rbac-proxy" not found Sep 09 23:17:35.552352 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.552326 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ld9zs_70c765f6-1f3d-4660-bc36-48293230f6e3/node-ca/0.log" Sep 09 23:17:35.858553 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.858474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:35.858553 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.858553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:35.858770 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.858653 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 23:17:35.858770 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.858676 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:17:35.858770 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.858693 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9597f89b-ng6z7: secret "image-registry-tls" not found Sep 09 23:17:35.858770 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.858730 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls podName:77b46604-88b1-439a-b69d-4689aa67e1f8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:51.858706483 +0000 UTC m=+65.671314840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-h422k" (UID: "77b46604-88b1-439a-b69d-4689aa67e1f8") : secret "samples-operator-tls" not found Sep 09 23:17:35.858770 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.858751 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls podName:c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:51.858740217 +0000 UTC m=+65.671348559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls") pod "image-registry-7f9597f89b-ng6z7" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8") : secret "image-registry-tls" not found Sep 09 23:17:35.932579 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.932539 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" event={"ID":"af5bcce7-094c-4f73-a233-d25bb0ecfb73","Type":"ContainerStarted","Data":"fee9b39f05cd5713a29b8ef0bc4581ced3dcc2a876b335bf201c8001785d0eb0"} Sep 09 23:17:35.959084 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.959053 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:35.959208 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.959193 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 09 23:17:35.959252 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.959206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:35.959252 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.959225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:35.959252 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.959241 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:51.959227745 +0000 UTC m=+65.771836089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : secret "router-metrics-certs-default" not found Sep 09 23:17:35.959393 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.959292 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:35.959393 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:35.959326 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:35.959393 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.959346 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle podName:da17354a-6849-474a-95c0-10c651c10d53 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:51.959326648 +0000 UTC m=+65.771935015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle") pod "router-default-f765574b6-292xd" (UID: "da17354a-6849-474a-95c0-10c651c10d53") : configmap references non-existent config key: service-ca.crt Sep 09 23:17:35.959393 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.959372 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:17:35.959529 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.959398 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls podName:7cc9c356-f4bd-409d-801a-d19df7559c6c nodeName:}" failed. No retries permitted until 2026-09-09 23:17:51.959383721 +0000 UTC m=+65.771992060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-kh9wq" (UID: "7cc9c356-f4bd-409d-801a-d19df7559c6c") : secret "cluster-monitoring-operator-tls" not found Sep 09 23:17:35.959529 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:35.959419 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls podName:6c5f9dc9-9bed-402b-af9f-16b918a054e5 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:51.959405558 +0000 UTC m=+65.772013898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls") pod "dns-default-8jpxs" (UID: "6c5f9dc9-9bed-402b-af9f-16b918a054e5") : secret "dns-default-metrics-tls" not found Sep 09 23:17:36.060781 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.060745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:36.060938 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:36.060907 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:17:36.060988 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:36.060970 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert podName:51a700c9-334f-4769-84f3-5e806b045aad nodeName:}" failed. No retries permitted until 2026-09-09 23:17:52.060955584 +0000 UTC m=+65.873563923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v5mr2" (UID: "51a700c9-334f-4769-84f3-5e806b045aad") : secret "networking-console-plugin-cert" not found Sep 09 23:17:36.259845 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.259814 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-nxrs9"] Sep 09 23:17:36.275622 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.275586 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-nxrs9"] Sep 09 23:17:36.275736 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.275697 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.278389 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.278368 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 09 23:17:36.279754 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.279734 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 09 23:17:36.279754 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.279750 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 09 23:17:36.279914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.279806 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zgwm5\"" Sep 09 23:17:36.279914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.279845 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 09 23:17:36.352001 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.351979 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4x4rk_6698e90f-06ad-4a2e-af70-ee5821fa0a0d/serve-healthcheck-canary/0.log" Sep 09 23:17:36.363149 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.363131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/65d6e1b5-da03-4a8a-964b-be811bfbaf84-signing-key\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.363237 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.363189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/65d6e1b5-da03-4a8a-964b-be811bfbaf84-signing-cabundle\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.363274 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.363241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffnvk\" (UniqueName: \"kubernetes.io/projected/65d6e1b5-da03-4a8a-964b-be811bfbaf84-kube-api-access-ffnvk\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.464108 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.464085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/65d6e1b5-da03-4a8a-964b-be811bfbaf84-signing-key\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.464186 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.464141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/65d6e1b5-da03-4a8a-964b-be811bfbaf84-signing-cabundle\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.464229 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.464183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffnvk\" (UniqueName: \"kubernetes.io/projected/65d6e1b5-da03-4a8a-964b-be811bfbaf84-kube-api-access-ffnvk\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.464778 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.464763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/65d6e1b5-da03-4a8a-964b-be811bfbaf84-signing-cabundle\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.466492 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.466468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/65d6e1b5-da03-4a8a-964b-be811bfbaf84-signing-key\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.471919 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.471891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffnvk\" (UniqueName: \"kubernetes.io/projected/65d6e1b5-da03-4a8a-964b-be811bfbaf84-kube-api-access-ffnvk\") pod \"service-ca-75d99c484-nxrs9\" (UID: \"65d6e1b5-da03-4a8a-964b-be811bfbaf84\") " pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.585523 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.585471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-nxrs9" Sep 09 23:17:36.939063 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.939033 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" event={"ID":"af5bcce7-094c-4f73-a233-d25bb0ecfb73","Type":"ContainerStarted","Data":"049d3e99aa8f6911b512bfc0db072151bf4960b1c74a887f5c38d3cbcea98832"} Sep 09 23:17:36.939369 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.939110 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" event={"ID":"af5bcce7-094c-4f73-a233-d25bb0ecfb73","Type":"ContainerStarted","Data":"1a1194164cf580405f93959d49790539343b9176f5a63f8bed8a9c79a2aab262"} Sep 09 23:17:36.951107 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.951084 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-nxrs9"] Sep 09 23:17:36.953543 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:36.953519 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d6e1b5_da03_4a8a_964b_be811bfbaf84.slice/crio-9f70b54e77f073a6eb6a9067610695faedd71fad11819aabdb04499c1714ad7f WatchSource:0}: Error finding container 9f70b54e77f073a6eb6a9067610695faedd71fad11819aabdb04499c1714ad7f: Status 404 returned error can't find the container with id 9f70b54e77f073a6eb6a9067610695faedd71fad11819aabdb04499c1714ad7f Sep 09 23:17:36.958840 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:36.958800 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-kd55b" podStartSLOduration=1.31442356 podStartE2EDuration="2.958785125s" totalImagesPullingTime="1.644361565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:36.957761844 +0000 UTC m=+50.770370202" watchObservedRunningTime="2026-09-09 23:17:36.958785125 +0000 UTC m=+50.771393487" Sep 09 23:17:37.270849 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:37.270815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:37.270849 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:37.270853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:37.271216 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:37.270964 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:17:37.271216 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:37.271015 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:41.271002524 +0000 UTC m=+55.083610877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : secret "insights-runtime-extractor-tls" not found Sep 09 23:17:37.271426 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:37.271400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42127793-921c-4220-b533-1797ac467275-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:37.947116 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:37.947076 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-nxrs9" event={"ID":"65d6e1b5-da03-4a8a-964b-be811bfbaf84","Type":"ContainerStarted","Data":"ffcf71e4685da5a50818c8dd51f84bde668b2fbd883e6d0bf83868eb8a472af5"} Sep 09 23:17:37.947573 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:37.947126 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-nxrs9" event={"ID":"65d6e1b5-da03-4a8a-964b-be811bfbaf84","Type":"ContainerStarted","Data":"9f70b54e77f073a6eb6a9067610695faedd71fad11819aabdb04499c1714ad7f"} Sep 09 23:17:37.964403 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:37.964350 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-nxrs9" podStartSLOduration=1.9643328169999998 podStartE2EDuration="1.964332817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:37.961469818 +0000 UTC m=+51.774078182" watchObservedRunningTime="2026-09-09 23:17:37.964332817 +0000 UTC m=+51.776941184" Sep 09 23:17:40.388190 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:40.388161 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:40.388627 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:40.388585 2559 scope.go:117] "RemoveContainer" containerID="130fc6af67d07d12b03ef35b3ce7afd9e8e72b82ea2b66bacb3233010b5f38d7" Sep 09 23:17:40.388823 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:40.388799 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-wq44d_openshift-console-operator(9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5)\"" pod="openshift-console-operator/console-operator-85b454874b-wq44d" podUID="9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5" Sep 09 23:17:40.389621 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:40.389581 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:40.961994 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:40.961957 2559 scope.go:117] "RemoveContainer" containerID="130fc6af67d07d12b03ef35b3ce7afd9e8e72b82ea2b66bacb3233010b5f38d7" Sep 09 23:17:40.962252 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:40.962166 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-wq44d_openshift-console-operator(9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5)\"" pod="openshift-console-operator/console-operator-85b454874b-wq44d" podUID="9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5" Sep 09 23:17:41.313538 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:41.313455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:41.313703 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:41.313652 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:17:41.313764 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:41.313732 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls podName:42127793-921c-4220-b533-1797ac467275 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:49.313710419 +0000 UTC m=+63.126318782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pf8kf" (UID: "42127793-921c-4220-b533-1797ac467275") : secret "insights-runtime-extractor-tls" not found Sep 09 23:17:41.967165 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:41.967125 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" event={"ID":"23398c6c-0e12-4d1e-992a-cd53788f7d86","Type":"ContainerStarted","Data":"b236433f9d8d4804a04ce113c4c7756d5fa3fdbb95f8450b65b55c9519b3ced3"} Sep 09 23:17:41.967608 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:41.967335 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:41.969147 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:41.969124 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" Sep 09 23:17:41.983905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:41.983861 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7bc56b8b8f-s9pwg" podStartSLOduration=10.350944744 podStartE2EDuration="30.983849115s" totalImagesPullingTime="20.632904371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:41.981878389 +0000 UTC m=+55.794486749" watchObservedRunningTime="2026-09-09 23:17:41.983849115 +0000 UTC m=+55.796457477" Sep 09 23:17:46.790396 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:46.790369 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b9zzg" Sep 09 23:17:49.386568 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:49.386517 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:49.388812 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:49.388790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42127793-921c-4220-b533-1797ac467275-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pf8kf\" (UID: \"42127793-921c-4220-b533-1797ac467275\") " pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:49.502274 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:49.502240 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnpcr\"" Sep 09 23:17:49.510368 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:49.510349 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pf8kf" Sep 09 23:17:49.775660 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:49.775630 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pf8kf"] Sep 09 23:17:49.778637 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:49.778335 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42127793_921c_4220_b533_1797ac467275.slice/crio-387b672d1c5348493c22ebe7f5b67b53d127105a3143117821136d344c4327e9 WatchSource:0}: Error finding container 387b672d1c5348493c22ebe7f5b67b53d127105a3143117821136d344c4327e9: Status 404 returned error can't find the container with id 387b672d1c5348493c22ebe7f5b67b53d127105a3143117821136d344c4327e9 Sep 09 23:17:49.996762 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:49.996730 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pf8kf" event={"ID":"42127793-921c-4220-b533-1797ac467275","Type":"ContainerStarted","Data":"ff222f042e61bde19df6d44b7354eddd1dc6bdd233911604a7e1bf784e479fef"} Sep 09 23:17:49.996922 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:49.996815 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pf8kf" event={"ID":"42127793-921c-4220-b533-1797ac467275","Type":"ContainerStarted","Data":"387b672d1c5348493c22ebe7f5b67b53d127105a3143117821136d344c4327e9"} Sep 09 23:17:51.638074 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:51.638049 2559 scope.go:117] "RemoveContainer" containerID="130fc6af67d07d12b03ef35b3ce7afd9e8e72b82ea2b66bacb3233010b5f38d7" Sep 09 23:17:51.914604 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:51.914504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:51.914604 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:51.914600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:51.917657 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:51.917628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77b46604-88b1-439a-b69d-4689aa67e1f8-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-h422k\" (UID: \"77b46604-88b1-439a-b69d-4689aa67e1f8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:51.917775 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:51.917638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"image-registry-7f9597f89b-ng6z7\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:52.005817 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.005782 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pf8kf" event={"ID":"42127793-921c-4220-b533-1797ac467275","Type":"ContainerStarted","Data":"539787fd6fe1d5f01f3f966092aa175ddd7f524c78d32b17c6f6b1f98e9c116d"} Sep 09 23:17:52.007545 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.007525 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:17:52.007659 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.007567 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-wq44d" event={"ID":"9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5","Type":"ContainerStarted","Data":"3ac19add570ad00640f28d64a1ede49edab0c6f85953bbab3278bbcddfbfe689"} Sep 09 23:17:52.007843 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.007826 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:52.015244 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.015217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:52.015390 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.015311 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:52.015390 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.015338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:52.015503 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.015415 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:52.016336 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.016204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da17354a-6849-474a-95c0-10c651c10d53-service-ca-bundle\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:52.018144 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.018121 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da17354a-6849-474a-95c0-10c651c10d53-metrics-certs\") pod \"router-default-f765574b6-292xd\" (UID: \"da17354a-6849-474a-95c0-10c651c10d53\") " pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:52.018871 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.018814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5f9dc9-9bed-402b-af9f-16b918a054e5-metrics-tls\") pod \"dns-default-8jpxs\" (UID: \"6c5f9dc9-9bed-402b-af9f-16b918a054e5\") " pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:52.018961 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.018903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cc9c356-f4bd-409d-801a-d19df7559c6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-kh9wq\" (UID: \"7cc9c356-f4bd-409d-801a-d19df7559c6c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:52.025672 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.025626 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-wq44d" podStartSLOduration=49.806777072 podStartE2EDuration="1m1.025608898s" totalImagesPullingTime="11.218831826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:52.024217711 +0000 UTC m=+65.836826074" watchObservedRunningTime="2026-09-09 23:17:52.025608898 +0000 UTC m=+65.838217261" Sep 09 23:17:52.116723 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.116687 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r67d2\"" Sep 09 23:17:52.116905 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.116767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:52.119061 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.119037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51a700c9-334f-4769-84f3-5e806b045aad-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v5mr2\" (UID: \"51a700c9-334f-4769-84f3-5e806b045aad\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:52.125349 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.125334 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:52.131408 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.131392 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-fwkgt\"" Sep 09 23:17:52.138943 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.138927 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" Sep 09 23:17:52.150707 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.150685 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-wq44d" Sep 09 23:17:52.161448 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.161423 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pr6dc\"" Sep 09 23:17:52.170364 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.170244 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" Sep 09 23:17:52.176112 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.175498 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kwbjl\"" Sep 09 23:17:52.185771 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.185665 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:52.199049 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.199028 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w7vc5\"" Sep 09 23:17:52.207875 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.207496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:52.338955 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.338931 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jdxq6\"" Sep 09 23:17:52.346405 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.346379 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" Sep 09 23:17:52.423217 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.423158 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:52.423381 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.423239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:52.426031 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.426007 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:17:52.426223 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.426006 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:17:52.429797 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.429776 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9597f89b-ng6z7"] Sep 09 23:17:52.431149 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:52.431126 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e7b0c6_a294_485c_9e6e_aef7cd1f9ed8.slice/crio-a1c5ba2de86c4b029b275c15aac2bcfc5274c0ed84a02d2490d5b4437e6cdcb0 WatchSource:0}: Error finding container a1c5ba2de86c4b029b275c15aac2bcfc5274c0ed84a02d2490d5b4437e6cdcb0: Status 404 returned error can't find the container with id a1c5ba2de86c4b029b275c15aac2bcfc5274c0ed84a02d2490d5b4437e6cdcb0 Sep 09 23:17:52.435398 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.435376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060a36f0-058c-4110-a094-dff084254fb5-metrics-certs\") pod \"network-metrics-daemon-ht64b\" (UID: \"060a36f0-058c-4110-a094-dff084254fb5\") " pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:52.435468 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.435401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a33b8ed-f089-4b62-b83c-23c8db52500b-original-pull-secret\") pod \"global-pull-secret-syncer-5rgz4\" (UID: \"1a33b8ed-f089-4b62-b83c-23c8db52500b\") " pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:52.467358 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.467336 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rgz4" Sep 09 23:17:52.481333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.481095 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k"] Sep 09 23:17:52.502741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.502716 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n28fd\"" Sep 09 23:17:52.510456 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.510396 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ht64b" Sep 09 23:17:52.513551 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.513299 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-f765574b6-292xd"] Sep 09 23:17:52.516573 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:52.516518 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda17354a_6849_474a_95c0_10c651c10d53.slice/crio-62fa02256a2f0eaf79113e6c0d4d51b6ad32c2865240449e00e243d086fcabbf WatchSource:0}: Error finding container 62fa02256a2f0eaf79113e6c0d4d51b6ad32c2865240449e00e243d086fcabbf: Status 404 returned error can't find the container with id 62fa02256a2f0eaf79113e6c0d4d51b6ad32c2865240449e00e243d086fcabbf Sep 09 23:17:52.535419 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.534536 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq"] Sep 09 23:17:52.536977 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.536680 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8jpxs"] Sep 09 23:17:52.537067 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:52.536996 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5f9dc9_9bed_402b_af9f_16b918a054e5.slice/crio-dcfd1ddc7743c3e3d1611b26879aa6fe292b41c136360657e9bc773752e39f7b WatchSource:0}: Error finding container dcfd1ddc7743c3e3d1611b26879aa6fe292b41c136360657e9bc773752e39f7b: Status 404 returned error can't find the container with id dcfd1ddc7743c3e3d1611b26879aa6fe292b41c136360657e9bc773752e39f7b Sep 09 23:17:52.626819 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.625854 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-v5mr2"] Sep 09 23:17:52.628338 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:52.628309 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a700c9_334f_4769_84f3_5e806b045aad.slice/crio-a8cfba4cfecca301d23c133fd744a92700887bc5f15182902f40667a3ede13bf WatchSource:0}: Error finding container a8cfba4cfecca301d23c133fd744a92700887bc5f15182902f40667a3ede13bf: Status 404 returned error can't find the container with id a8cfba4cfecca301d23c133fd744a92700887bc5f15182902f40667a3ede13bf Sep 09 23:17:52.750494 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.749685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rgz4"] Sep 09 23:17:52.804705 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:52.804554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ht64b"] Sep 09 23:17:53.013970 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.013885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" event={"ID":"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8","Type":"ContainerStarted","Data":"86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd"} Sep 09 23:17:53.013970 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.013941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" event={"ID":"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8","Type":"ContainerStarted","Data":"a1c5ba2de86c4b029b275c15aac2bcfc5274c0ed84a02d2490d5b4437e6cdcb0"} Sep 09 23:17:53.014318 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.013999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:17:53.015335 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.015303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" event={"ID":"7cc9c356-f4bd-409d-801a-d19df7559c6c","Type":"ContainerStarted","Data":"ee1c3b4965298507aa23f7cc53ebd40d728c6cc8432f5e8c82c378f30d53269d"} Sep 09 23:17:53.017035 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.016957 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f765574b6-292xd" event={"ID":"da17354a-6849-474a-95c0-10c651c10d53","Type":"ContainerStarted","Data":"a715358d6476a66a38dc69f1f65c2dbf6663cc074997d35386496996c0a80028"} Sep 09 23:17:53.017035 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.016998 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-f765574b6-292xd" event={"ID":"da17354a-6849-474a-95c0-10c651c10d53","Type":"ContainerStarted","Data":"62fa02256a2f0eaf79113e6c0d4d51b6ad32c2865240449e00e243d086fcabbf"} Sep 09 23:17:53.019096 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.019071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" event={"ID":"51a700c9-334f-4769-84f3-5e806b045aad","Type":"ContainerStarted","Data":"a8cfba4cfecca301d23c133fd744a92700887bc5f15182902f40667a3ede13bf"} Sep 09 23:17:53.020193 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.020168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jpxs" event={"ID":"6c5f9dc9-9bed-402b-af9f-16b918a054e5","Type":"ContainerStarted","Data":"dcfd1ddc7743c3e3d1611b26879aa6fe292b41c136360657e9bc773752e39f7b"} Sep 09 23:17:53.021292 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.021268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" event={"ID":"77b46604-88b1-439a-b69d-4689aa67e1f8","Type":"ContainerStarted","Data":"5c70f7ace4d5f46232193ad5de39434c226e4d3d1246217bc3f3250ecfd1ee94"} Sep 09 23:17:53.033078 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.032880 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" podStartSLOduration=66.032865598 podStartE2EDuration="1m6.032865598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:53.032862688 +0000 UTC m=+66.845471085" watchObservedRunningTime="2026-09-09 23:17:53.032865598 +0000 UTC m=+66.845473963" Sep 09 23:17:53.033792 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:53.033771 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060a36f0_058c_4110_a094_dff084254fb5.slice/crio-73d728ffbad78bf985ebad0f0e3ff693c666cdae98882ab43bfe9928ea4de412 WatchSource:0}: Error finding container 73d728ffbad78bf985ebad0f0e3ff693c666cdae98882ab43bfe9928ea4de412: Status 404 returned error can't find the container with id 73d728ffbad78bf985ebad0f0e3ff693c666cdae98882ab43bfe9928ea4de412 Sep 09 23:17:53.034648 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:53.034629 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a33b8ed_f089_4b62_b83c_23c8db52500b.slice/crio-aa4c35a07eb3789a4dd4d36e6e9cec6dc89810f51dd24d4acf515686fcc11c1d WatchSource:0}: Error finding container aa4c35a07eb3789a4dd4d36e6e9cec6dc89810f51dd24d4acf515686fcc11c1d: Status 404 returned error can't find the container with id aa4c35a07eb3789a4dd4d36e6e9cec6dc89810f51dd24d4acf515686fcc11c1d Sep 09 23:17:53.051004 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.050968 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-f765574b6-292xd" podStartSLOduration=62.050956289 podStartE2EDuration="1m2.050956289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:53.050447458 +0000 UTC m=+66.863055844" watchObservedRunningTime="2026-09-09 23:17:53.050956289 +0000 UTC m=+66.863564648" Sep 09 23:17:53.186767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.186740 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:53.189208 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:53.189190 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:54.031035 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:54.030999 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ht64b" event={"ID":"060a36f0-058c-4110-a094-dff084254fb5","Type":"ContainerStarted","Data":"73d728ffbad78bf985ebad0f0e3ff693c666cdae98882ab43bfe9928ea4de412"} Sep 09 23:17:54.033108 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:54.033036 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rgz4" event={"ID":"1a33b8ed-f089-4b62-b83c-23c8db52500b","Type":"ContainerStarted","Data":"aa4c35a07eb3789a4dd4d36e6e9cec6dc89810f51dd24d4acf515686fcc11c1d"} Sep 09 23:17:54.039919 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:54.039274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pf8kf" event={"ID":"42127793-921c-4220-b533-1797ac467275","Type":"ContainerStarted","Data":"3078fa65856cdc9ac563950630dcd0a5ad1eab97a9333d694f342509f3da413e"} Sep 09 23:17:54.039919 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:54.039328 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:54.041155 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:54.041116 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-f765574b6-292xd" Sep 09 23:17:54.057529 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:54.056802 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pf8kf" podStartSLOduration=17.953207371 podStartE2EDuration="21.056783091s" totalImagesPullingTime="3.10357572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:17:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:54.056513434 +0000 UTC m=+67.869121802" watchObservedRunningTime="2026-09-09 23:17:54.056783091 +0000 UTC m=+67.869391452" Sep 09 23:17:55.702861 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.702512 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-xb6wn"] Sep 09 23:17:55.709429 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.709396 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-xb6wn" Sep 09 23:17:55.712739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.712436 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hvlsq\"" Sep 09 23:17:55.712739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.712624 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 23:17:55.712739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.712685 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 23:17:55.717503 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.717481 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-xb6wn"] Sep 09 23:17:55.859119 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.859083 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9sz\" (UniqueName: \"kubernetes.io/projected/b1927131-fc1c-4fe7-a392-8400e1d6b6b7-kube-api-access-9f9sz\") pod \"downloads-85d98c998f-xb6wn\" (UID: \"b1927131-fc1c-4fe7-a392-8400e1d6b6b7\") " pod="openshift-console/downloads-85d98c998f-xb6wn" Sep 09 23:17:55.960039 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.959972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9sz\" (UniqueName: \"kubernetes.io/projected/b1927131-fc1c-4fe7-a392-8400e1d6b6b7-kube-api-access-9f9sz\") pod \"downloads-85d98c998f-xb6wn\" (UID: \"b1927131-fc1c-4fe7-a392-8400e1d6b6b7\") " pod="openshift-console/downloads-85d98c998f-xb6wn" Sep 09 23:17:55.974197 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:55.974169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9sz\" (UniqueName: \"kubernetes.io/projected/b1927131-fc1c-4fe7-a392-8400e1d6b6b7-kube-api-access-9f9sz\") pod \"downloads-85d98c998f-xb6wn\" (UID: \"b1927131-fc1c-4fe7-a392-8400e1d6b6b7\") " pod="openshift-console/downloads-85d98c998f-xb6wn" Sep 09 23:17:56.026226 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.026193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-xb6wn" Sep 09 23:17:56.541248 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.540987 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-xb6wn"] Sep 09 23:17:56.548309 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:56.548271 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1927131_fc1c_4fe7_a392_8400e1d6b6b7.slice/crio-cc604255d2841372cceebb3bdb3149cd3d462140413af47a6ed37dd2e2a898ea WatchSource:0}: Error finding container cc604255d2841372cceebb3bdb3149cd3d462140413af47a6ed37dd2e2a898ea: Status 404 returned error can't find the container with id cc604255d2841372cceebb3bdb3149cd3d462140413af47a6ed37dd2e2a898ea Sep 09 23:17:56.846361 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.846253 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp"] Sep 09 23:17:56.850856 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.850832 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:17:56.854185 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.853764 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 23:17:56.854185 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.854003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nwk7m\"" Sep 09 23:17:56.858291 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.858203 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp"] Sep 09 23:17:56.968802 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:56.968765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bf26d86-df94-4685-8519-92a5fa9dd1f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dfwzp\" (UID: \"8bf26d86-df94-4685-8519-92a5fa9dd1f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:17:57.054408 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.054372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" event={"ID":"7cc9c356-f4bd-409d-801a-d19df7559c6c","Type":"ContainerStarted","Data":"5ec88f442cd56f841766569839b087e7737f59b2b67b6418a4855328f34dcba7"} Sep 09 23:17:57.056866 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.056834 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ht64b" event={"ID":"060a36f0-058c-4110-a094-dff084254fb5","Type":"ContainerStarted","Data":"9672eeba734975b24fb56e09eef098ad12a99fc999d40035799c87f751f64808"} Sep 09 23:17:57.057009 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.056894 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ht64b" event={"ID":"060a36f0-058c-4110-a094-dff084254fb5","Type":"ContainerStarted","Data":"080519fbcac5b4fb08222eb378dcb066fd32619e293ad5e5a1994cd5630219d9"} Sep 09 23:17:57.059612 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.059034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-xb6wn" event={"ID":"b1927131-fc1c-4fe7-a392-8400e1d6b6b7","Type":"ContainerStarted","Data":"cc604255d2841372cceebb3bdb3149cd3d462140413af47a6ed37dd2e2a898ea"} Sep 09 23:17:57.062327 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.061569 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" event={"ID":"51a700c9-334f-4769-84f3-5e806b045aad","Type":"ContainerStarted","Data":"0fe59cc8afb599f864c6947a5086c45f22f179eebeba38ec96529b565210627b"} Sep 09 23:17:57.070401 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.070373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bf26d86-df94-4685-8519-92a5fa9dd1f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dfwzp\" (UID: \"8bf26d86-df94-4685-8519-92a5fa9dd1f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:17:57.070656 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:57.070635 2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 09 23:17:57.070741 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:17:57.070726 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf26d86-df94-4685-8519-92a5fa9dd1f3-tls-certificates podName:8bf26d86-df94-4685-8519-92a5fa9dd1f3 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:57.570699107 +0000 UTC m=+71.383307461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8bf26d86-df94-4685-8519-92a5fa9dd1f3-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-dfwzp" (UID: "8bf26d86-df94-4685-8519-92a5fa9dd1f3") : secret "prometheus-operator-admission-webhook-tls" not found Sep 09 23:17:57.071609 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.071566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jpxs" event={"ID":"6c5f9dc9-9bed-402b-af9f-16b918a054e5","Type":"ContainerStarted","Data":"5e5ac779d19c8efa1c275803cf324955c75ba61086b0c6dbb6ace3739225db7a"} Sep 09 23:17:57.071696 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.071673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8jpxs" event={"ID":"6c5f9dc9-9bed-402b-af9f-16b918a054e5","Type":"ContainerStarted","Data":"4c276b9e085028561f3f398d75df75e7fd713de3229e5c2c9346677b39d40698"} Sep 09 23:17:57.071846 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.071823 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8jpxs" Sep 09 23:17:57.074959 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.074935 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" event={"ID":"77b46604-88b1-439a-b69d-4689aa67e1f8","Type":"ContainerStarted","Data":"190bdd6ef674fddb26626765fc74afac4d76d974d9b7785392309c3c3f00cacb"} Sep 09 23:17:57.075194 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.074983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" event={"ID":"77b46604-88b1-439a-b69d-4689aa67e1f8","Type":"ContainerStarted","Data":"d6e30b497146afd9d663e7c0858cbd7f9269b9b4a67b00d48d5ea367dae0ec9c"} Sep 09 23:17:57.076568 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.076516 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-kh9wq" podStartSLOduration=62.364682346 podStartE2EDuration="1m6.076498669s" totalImagesPullingTime="3.711816323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:57.07379285 +0000 UTC m=+70.886401217" watchObservedRunningTime="2026-09-09 23:17:57.076498669 +0000 UTC m=+70.889107038" Sep 09 23:17:57.092844 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.092805 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ht64b" podStartSLOduration=66.871652193 podStartE2EDuration="1m10.092790343s" totalImagesPullingTime="3.22113815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:57.090882296 +0000 UTC m=+70.903490659" watchObservedRunningTime="2026-09-09 23:17:57.092790343 +0000 UTC m=+70.905398706" Sep 09 23:17:57.107501 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.107419 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-v5mr2" podStartSLOduration=33.480691608 podStartE2EDuration="37.107404722s" totalImagesPullingTime="3.626713114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:57.106668152 +0000 UTC m=+70.919276517" watchObservedRunningTime="2026-09-09 23:17:57.107404722 +0000 UTC m=+70.920013083" Sep 09 23:17:57.122939 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.122896 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8jpxs" podStartSLOduration=34.407805181 podStartE2EDuration="38.122883727s" totalImagesPullingTime="3.715078546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:57.122090341 +0000 UTC m=+70.934698704" watchObservedRunningTime="2026-09-09 23:17:57.122883727 +0000 UTC m=+70.935492106" Sep 09 23:17:57.139383 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.139303 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-h422k" podStartSLOduration=62.394094877 podStartE2EDuration="1m6.139286357s" totalImagesPullingTime="3.74519148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:57.137950736 +0000 UTC m=+70.950559100" watchObservedRunningTime="2026-09-09 23:17:57.139286357 +0000 UTC m=+70.951894725" Sep 09 23:17:57.576579 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.576537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bf26d86-df94-4685-8519-92a5fa9dd1f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dfwzp\" (UID: \"8bf26d86-df94-4685-8519-92a5fa9dd1f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:17:57.579735 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.579703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8bf26d86-df94-4685-8519-92a5fa9dd1f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dfwzp\" (UID: \"8bf26d86-df94-4685-8519-92a5fa9dd1f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:17:57.766305 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:57.766271 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:17:58.461721 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:58.461697 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp"] Sep 09 23:17:58.557848 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:17:58.557819 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf26d86_df94_4685_8519_92a5fa9dd1f3.slice/crio-7360029d0d5a7fb6a8e2e57796a099ef944b1ad25bf08ae35c168312f2ecb6a0 WatchSource:0}: Error finding container 7360029d0d5a7fb6a8e2e57796a099ef944b1ad25bf08ae35c168312f2ecb6a0: Status 404 returned error can't find the container with id 7360029d0d5a7fb6a8e2e57796a099ef944b1ad25bf08ae35c168312f2ecb6a0 Sep 09 23:17:59.086030 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:59.085944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" event={"ID":"8bf26d86-df94-4685-8519-92a5fa9dd1f3","Type":"ContainerStarted","Data":"7360029d0d5a7fb6a8e2e57796a099ef944b1ad25bf08ae35c168312f2ecb6a0"} Sep 09 23:17:59.087627 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:59.087583 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rgz4" event={"ID":"1a33b8ed-f089-4b62-b83c-23c8db52500b","Type":"ContainerStarted","Data":"de268aa53b6c4fa8c95237d656b7c3b6b8dcd15e2a4d041b30af2eca5050ae13"} Sep 09 23:17:59.103039 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:17:59.102982 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5rgz4" podStartSLOduration=66.537390949 podStartE2EDuration="1m12.102966862s" totalImagesPullingTime="5.565575913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:59.101727696 +0000 UTC m=+72.914336062" watchObservedRunningTime="2026-09-09 23:17:59.102966862 +0000 UTC m=+72.915575223" Sep 09 23:18:01.101726 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.101689 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" event={"ID":"8bf26d86-df94-4685-8519-92a5fa9dd1f3","Type":"ContainerStarted","Data":"e14e8266b7d95651a1d5cd5205231e0ba1071a0f8445c68e470c312acb2cde9c"} Sep 09 23:18:01.102252 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.102231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:18:01.107646 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.107618 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" Sep 09 23:18:01.139390 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.139336 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dfwzp" podStartSLOduration=3.488641478 podStartE2EDuration="5.13931653s" totalImagesPullingTime="1.650675052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:01.134080176 +0000 UTC m=+74.946688560" watchObservedRunningTime="2026-09-09 23:18:01.13931653 +0000 UTC m=+74.951924894" Sep 09 23:18:01.253947 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.253912 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56c585d759-jmg2l"] Sep 09 23:18:01.286004 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.285969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56c585d759-jmg2l"] Sep 09 23:18:01.286157 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.286090 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.288964 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.288822 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 23:18:01.288964 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.288844 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 23:18:01.288964 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.288859 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 23:18:01.288964 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.288894 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7t5s\"" Sep 09 23:18:01.288964 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.288860 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 23:18:01.289390 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.289360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 23:18:01.306574 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.306550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-config\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.306683 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.306615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-service-ca\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.306683 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.306647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-serving-cert\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.306683 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.306671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-oauth-serving-cert\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.306804 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.306779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c645m\" (UniqueName: \"kubernetes.io/projected/85a67da6-2beb-4f01-bc0c-ae50d48783a1-kube-api-access-c645m\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.306853 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.306813 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-oauth-config\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.407309 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.407234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-service-ca\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.407309 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.407276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-serving-cert\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.407309 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.407302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-oauth-serving-cert\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.407906 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.407572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c645m\" (UniqueName: \"kubernetes.io/projected/85a67da6-2beb-4f01-bc0c-ae50d48783a1-kube-api-access-c645m\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.407906 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.407653 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-oauth-config\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.407906 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.407711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-config\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.408126 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.408078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-service-ca\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.408126 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.408103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-oauth-serving-cert\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.408297 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.408277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-config\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.410010 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.409985 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-serving-cert\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.410243 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.410224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-oauth-config\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.415493 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.415473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c645m\" (UniqueName: \"kubernetes.io/projected/85a67da6-2beb-4f01-bc0c-ae50d48783a1-kube-api-access-c645m\") pod \"console-56c585d759-jmg2l\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.597361 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.597326 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:01.910143 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:01.909321 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56c585d759-jmg2l"] Sep 09 23:18:01.912275 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:18:01.912237 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a67da6_2beb_4f01_bc0c_ae50d48783a1.slice/crio-190dc87398cbbc02b4c8685674d622ce650af2036ad77fbfeb394eb0b84651eb WatchSource:0}: Error finding container 190dc87398cbbc02b4c8685674d622ce650af2036ad77fbfeb394eb0b84651eb: Status 404 returned error can't find the container with id 190dc87398cbbc02b4c8685674d622ce650af2036ad77fbfeb394eb0b84651eb Sep 09 23:18:02.108637 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:02.108537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c585d759-jmg2l" event={"ID":"85a67da6-2beb-4f01-bc0c-ae50d48783a1","Type":"ContainerStarted","Data":"190dc87398cbbc02b4c8685674d622ce650af2036ad77fbfeb394eb0b84651eb"} Sep 09 23:18:03.925518 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:03.925410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bdjlj" Sep 09 23:18:07.082720 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:07.082684 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8jpxs" Sep 09 23:18:07.140463 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:07.140427 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c585d759-jmg2l" event={"ID":"85a67da6-2beb-4f01-bc0c-ae50d48783a1","Type":"ContainerStarted","Data":"abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce"} Sep 09 23:18:07.156516 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:07.156444 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56c585d759-jmg2l" podStartSLOduration=2.000390628 podStartE2EDuration="6.156423068s" totalImagesPullingTime="4.15603244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:07.155823552 +0000 UTC m=+80.968431915" watchObservedRunningTime="2026-09-09 23:18:07.156423068 +0000 UTC m=+80.969031434" Sep 09 23:18:09.295736 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.295213 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cxzrg"] Sep 09 23:18:09.298774 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.298753 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.301924 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.301504 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bt9rk\"" Sep 09 23:18:09.301924 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.301739 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 23:18:09.301924 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.301755 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 23:18:09.302430 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.302238 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 23:18:09.302766 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.302711 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 23:18:09.379392 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379313 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379392 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379369 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-textfile\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379578 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-wtmp\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379578 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e12116-c427-4942-893d-ef88d9af2f06-metrics-client-ca\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379738 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379574 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379738 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-root\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379738 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75r2\" (UniqueName: \"kubernetes.io/projected/e5e12116-c427-4942-893d-ef88d9af2f06-kube-api-access-j75r2\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379738 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-sys\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.379738 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.379704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481134 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481160 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-textfile\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-wtmp\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e12116-c427-4942-893d-ef88d9af2f06-metrics-client-ca\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-root\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481645 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75r2\" (UniqueName: \"kubernetes.io/projected/e5e12116-c427-4942-893d-ef88d9af2f06-kube-api-access-j75r2\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481645 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481335 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-sys\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481645 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.481645 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:18:09.481503 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 23:18:09.481645 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:18:09.481578 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls podName:e5e12116-c427-4942-893d-ef88d9af2f06 nodeName:}" failed. No retries permitted until 2026-09-09 23:18:09.981558111 +0000 UTC m=+83.794166453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls") pod "node-exporter-cxzrg" (UID: "e5e12116-c427-4942-893d-ef88d9af2f06") : secret "node-exporter-tls" not found Sep 09 23:18:09.481645 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-textfile\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.482349 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.481957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-wtmp\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.482349 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.482265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-sys\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.482349 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.482315 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5e12116-c427-4942-893d-ef88d9af2f06-root\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.482621 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.482483 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e12116-c427-4942-893d-ef88d9af2f06-metrics-client-ca\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.482831 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.482809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.484807 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.484785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.490971 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.490925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75r2\" (UniqueName: \"kubernetes.io/projected/e5e12116-c427-4942-893d-ef88d9af2f06-kube-api-access-j75r2\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.986709 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:09.986641 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:09.986877 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:18:09.986837 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 23:18:09.986953 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:18:09.986930 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls podName:e5e12116-c427-4942-893d-ef88d9af2f06 nodeName:}" failed. No retries permitted until 2026-09-09 23:18:10.986908842 +0000 UTC m=+84.799517206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls") pod "node-exporter-cxzrg" (UID: "e5e12116-c427-4942-893d-ef88d9af2f06") : secret "node-exporter-tls" not found Sep 09 23:18:10.997483 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:10.997450 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:11.000106 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:11.000076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e12116-c427-4942-893d-ef88d9af2f06-node-exporter-tls\") pod \"node-exporter-cxzrg\" (UID: \"e5e12116-c427-4942-893d-ef88d9af2f06\") " pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:11.112508 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:11.112475 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxzrg" Sep 09 23:18:11.598261 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:11.598227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:11.598453 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:11.598296 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:11.603741 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:11.603720 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:12.130876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:12.130828 2559 patch_prober.go:28] interesting pod/image-registry-7f9597f89b-ng6z7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 23:18:12.131515 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:12.130900 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" podUID="c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:18:12.181655 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:12.181629 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:14.045121 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:14.044836 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:18:15.302863 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:18:15.302832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e12116_c427_4942_893d_ef88d9af2f06.slice/crio-f5a8a305f923ef0a0e013e31edbbbf6d7299c2ea122cb7d068c3501967a4f32a WatchSource:0}: Error finding container f5a8a305f923ef0a0e013e31edbbbf6d7299c2ea122cb7d068c3501967a4f32a: Status 404 returned error can't find the container with id f5a8a305f923ef0a0e013e31edbbbf6d7299c2ea122cb7d068c3501967a4f32a Sep 09 23:18:16.204829 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:16.204785 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxzrg" event={"ID":"e5e12116-c427-4942-893d-ef88d9af2f06","Type":"ContainerStarted","Data":"f5a8a305f923ef0a0e013e31edbbbf6d7299c2ea122cb7d068c3501967a4f32a"} Sep 09 23:18:16.206835 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:16.206806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-xb6wn" event={"ID":"b1927131-fc1c-4fe7-a392-8400e1d6b6b7","Type":"ContainerStarted","Data":"6bff656738d097d8e294c18d77bff30f9892779596007e0afdb1ba0b2759a1e4"} Sep 09 23:18:16.207068 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:16.207028 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-xb6wn" Sep 09 23:18:16.216185 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:16.216166 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-xb6wn" Sep 09 23:18:16.229537 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:16.229485 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-xb6wn" podStartSLOduration=2.08197406 podStartE2EDuration="21.229466841s" totalImagesPullingTime="19.147492781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:16.228087981 +0000 UTC m=+90.040696344" watchObservedRunningTime="2026-09-09 23:18:16.229466841 +0000 UTC m=+90.042075204" Sep 09 23:18:17.215278 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:17.215233 2559 generic.go:356] "Generic (PLEG): container finished" podID="e5e12116-c427-4942-893d-ef88d9af2f06" containerID="d7e1481b574c79532dc8c64231e816b39dcdc56d317f5a099ae7ef8a42f5d035" exitCode=0 Sep 09 23:18:17.215767 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:17.215337 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxzrg" event={"ID":"e5e12116-c427-4942-893d-ef88d9af2f06","Type":"ContainerDied","Data":"d7e1481b574c79532dc8c64231e816b39dcdc56d317f5a099ae7ef8a42f5d035"} Sep 09 23:18:18.229763 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:18.229723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxzrg" event={"ID":"e5e12116-c427-4942-893d-ef88d9af2f06","Type":"ContainerStarted","Data":"bfc5d8b4e1592f786502c7b0c0457f3913f1a28c15a2311ebdf2e413c427266c"} Sep 09 23:18:18.230207 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:18.229819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxzrg" event={"ID":"e5e12116-c427-4942-893d-ef88d9af2f06","Type":"ContainerStarted","Data":"2bfdb4208e8c537efa5dba460090d78db50910e7b706031245be03cfccc8a565"} Sep 09 23:18:18.252823 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:18.252776 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cxzrg" podStartSLOduration=8.270163863 podStartE2EDuration="9.252762274s" totalImagesPullingTime="906.340238ms" totalInitContainerRuntime="76.258173ms" isStatefulPod=true podCreationTimestamp="2026-09-09 23:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:18.251151085 +0000 UTC m=+92.063759448" watchObservedRunningTime="2026-09-09 23:18:18.252762274 +0000 UTC m=+92.065370636" Sep 09 23:18:24.407032 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:24.406999 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9597f89b-ng6z7"] Sep 09 23:18:26.528064 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:26.528018 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56c585d759-jmg2l"] Sep 09 23:18:48.364696 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:48.364668 2559 generic.go:356] "Generic (PLEG): container finished" podID="41109807-ec62-4222-ab2b-43a068961aab" containerID="d50cda0e0aa1b2979eee28cb9567d01b28d63789eb2641ea2a0250296b14701a" exitCode=0 Sep 09 23:18:48.365050 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:48.364738 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" event={"ID":"41109807-ec62-4222-ab2b-43a068961aab","Type":"ContainerDied","Data":"d50cda0e0aa1b2979eee28cb9567d01b28d63789eb2641ea2a0250296b14701a"} Sep 09 23:18:48.365050 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:48.365023 2559 scope.go:117] "RemoveContainer" containerID="d50cda0e0aa1b2979eee28cb9567d01b28d63789eb2641ea2a0250296b14701a" Sep 09 23:18:49.371469 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.371435 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-j8vqn" event={"ID":"41109807-ec62-4222-ab2b-43a068961aab","Type":"ContainerStarted","Data":"16b7954d40d2ed09ec3fb9bbe943fc48de41f75bb75d6d7f9aa808bf673643dd"} Sep 09 23:18:49.431841 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.431780 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" podUID="c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" containerName="registry" containerID="cri-o://86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd" gracePeriod=30 Sep 09 23:18:49.717940 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.717915 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:18:49.826611 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.826566 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-certificates\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.826739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.826620 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-kube-api-access-2xd2z\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.826739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.826654 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-ca-trust-extracted\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.826739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.826681 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-installation-pull-secrets\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.826739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.826699 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-trusted-ca\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.826739 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.826714 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.827048 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.826749 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-bound-sa-token\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.827106 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.827082 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-image-registry-private-configuration\") pod \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\" (UID: \"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8\") " Sep 09 23:18:49.827160 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.827098 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:18:49.827358 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.827246 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:18:49.828463 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.827493 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-trusted-ca\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:49.828463 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.827517 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-certificates\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:49.829698 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.829648 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:18:49.829698 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.829653 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:18:49.829868 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.829773 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-kube-api-access-2xd2z" (OuterVolumeSpecName: "kube-api-access-2xd2z") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "kube-api-access-2xd2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:18:49.829868 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.829799 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:18:49.829868 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.829821 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:18:49.835584 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.835562 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" (UID: "c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:18:49.928165 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.928102 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-installation-pull-secrets\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:49.928165 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.928129 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-registry-tls\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:49.928165 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.928141 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-bound-sa-token\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:49.928165 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.928150 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-image-registry-private-configuration\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:49.928165 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.928158 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xd2z\" (UniqueName: \"kubernetes.io/projected/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-kube-api-access-2xd2z\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:49.928391 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:49.928170 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8-ca-trust-extracted\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:50.382216 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.382137 2559 generic.go:356] "Generic (PLEG): container finished" podID="c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" containerID="86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd" exitCode=0 Sep 09 23:18:50.382216 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.382178 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" event={"ID":"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8","Type":"ContainerDied","Data":"86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd"} Sep 09 23:18:50.382216 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.382195 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" Sep 09 23:18:50.382749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.382238 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9597f89b-ng6z7" event={"ID":"c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8","Type":"ContainerDied","Data":"a1c5ba2de86c4b029b275c15aac2bcfc5274c0ed84a02d2490d5b4437e6cdcb0"} Sep 09 23:18:50.382749 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.382257 2559 scope.go:117] "RemoveContainer" containerID="86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd" Sep 09 23:18:50.391523 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.391506 2559 scope.go:117] "RemoveContainer" containerID="86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd" Sep 09 23:18:50.391882 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:18:50.391856 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd\": container with ID starting with 86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd not found: ID does not exist" containerID="86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd" Sep 09 23:18:50.391944 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.391894 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd"} err="failed to get container status \"86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd\": rpc error: code = NotFound desc = could not find container \"86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd\": container with ID starting with 86f74c70cf97fbe47489265d7eec0d6666f1892c945c2157a86e26415f3e17cd not found: ID does not exist" Sep 09 23:18:50.403675 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.403647 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9597f89b-ng6z7"] Sep 09 23:18:50.408787 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.408767 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f9597f89b-ng6z7"] Sep 09 23:18:50.485903 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.485873 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" podUID="67d1f8cf-5d76-4d43-b63c-97933be1a863" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 23:18:50.643586 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:50.643513 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" path="/var/lib/kubelet/pods/c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8/volumes" Sep 09 23:18:51.553078 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.553023 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56c585d759-jmg2l" podUID="85a67da6-2beb-4f01-bc0c-ae50d48783a1" containerName="console" containerID="cri-o://abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce" gracePeriod=15 Sep 09 23:18:51.833554 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.833498 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:51.845370 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845349 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-serving-cert\") pod \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " Sep 09 23:18:51.845442 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845391 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-oauth-config\") pod \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " Sep 09 23:18:51.845442 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845415 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-oauth-serving-cert\") pod \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " Sep 09 23:18:51.845442 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845431 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c645m\" (UniqueName: \"kubernetes.io/projected/85a67da6-2beb-4f01-bc0c-ae50d48783a1-kube-api-access-c645m\") pod \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " Sep 09 23:18:51.845554 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845490 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-config\") pod \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " Sep 09 23:18:51.845554 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845509 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-service-ca\") pod \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\" (UID: \"85a67da6-2beb-4f01-bc0c-ae50d48783a1\") " Sep 09 23:18:51.845854 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845822 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "85a67da6-2beb-4f01-bc0c-ae50d48783a1" (UID: "85a67da6-2beb-4f01-bc0c-ae50d48783a1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:18:51.845945 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845846 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-config" (OuterVolumeSpecName: "console-config") pod "85a67da6-2beb-4f01-bc0c-ae50d48783a1" (UID: "85a67da6-2beb-4f01-bc0c-ae50d48783a1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:18:51.845945 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.845888 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-service-ca" (OuterVolumeSpecName: "service-ca") pod "85a67da6-2beb-4f01-bc0c-ae50d48783a1" (UID: "85a67da6-2beb-4f01-bc0c-ae50d48783a1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:18:51.847581 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.847563 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "85a67da6-2beb-4f01-bc0c-ae50d48783a1" (UID: "85a67da6-2beb-4f01-bc0c-ae50d48783a1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:18:51.847665 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.847584 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a67da6-2beb-4f01-bc0c-ae50d48783a1-kube-api-access-c645m" (OuterVolumeSpecName: "kube-api-access-c645m") pod "85a67da6-2beb-4f01-bc0c-ae50d48783a1" (UID: "85a67da6-2beb-4f01-bc0c-ae50d48783a1"). InnerVolumeSpecName "kube-api-access-c645m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:18:51.847711 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.847669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "85a67da6-2beb-4f01-bc0c-ae50d48783a1" (UID: "85a67da6-2beb-4f01-bc0c-ae50d48783a1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:18:51.946805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.946779 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-oauth-config\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:51.946805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.946801 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-oauth-serving-cert\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:51.946944 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.946810 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c645m\" (UniqueName: \"kubernetes.io/projected/85a67da6-2beb-4f01-bc0c-ae50d48783a1-kube-api-access-c645m\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:51.946944 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.946819 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-config\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:51.946944 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.946831 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85a67da6-2beb-4f01-bc0c-ae50d48783a1-service-ca\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:51.946944 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:51.946838 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85a67da6-2beb-4f01-bc0c-ae50d48783a1-console-serving-cert\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:18:52.392473 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.392438 2559 generic.go:356] "Generic (PLEG): container finished" podID="85a67da6-2beb-4f01-bc0c-ae50d48783a1" containerID="abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce" exitCode=0 Sep 09 23:18:52.392648 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.392478 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56c585d759-jmg2l" Sep 09 23:18:52.392648 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.392522 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c585d759-jmg2l" event={"ID":"85a67da6-2beb-4f01-bc0c-ae50d48783a1","Type":"ContainerDied","Data":"abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce"} Sep 09 23:18:52.392648 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.392573 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56c585d759-jmg2l" event={"ID":"85a67da6-2beb-4f01-bc0c-ae50d48783a1","Type":"ContainerDied","Data":"190dc87398cbbc02b4c8685674d622ce650af2036ad77fbfeb394eb0b84651eb"} Sep 09 23:18:52.392648 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.392608 2559 scope.go:117] "RemoveContainer" containerID="abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce" Sep 09 23:18:52.401158 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.401144 2559 scope.go:117] "RemoveContainer" containerID="abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce" Sep 09 23:18:52.401441 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:18:52.401421 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce\": container with ID starting with abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce not found: ID does not exist" containerID="abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce" Sep 09 23:18:52.401486 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.401450 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce"} err="failed to get container status \"abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce\": rpc error: code = NotFound desc = could not find container \"abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce\": container with ID starting with abd19495769eee879dcacd3347d5ff8392a7fa06f465d8998b8f01df5dac9fce not found: ID does not exist" Sep 09 23:18:52.413134 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.413110 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56c585d759-jmg2l"] Sep 09 23:18:52.416527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.416485 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56c585d759-jmg2l"] Sep 09 23:18:52.642834 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:52.642757 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a67da6-2beb-4f01-bc0c-ae50d48783a1" path="/var/lib/kubelet/pods/85a67da6-2beb-4f01-bc0c-ae50d48783a1/volumes" Sep 09 23:18:53.398966 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:53.398943 2559 generic.go:356] "Generic (PLEG): container finished" podID="9eea67d0-b1b9-4305-811e-20aa6ea233ed" containerID="b4f8ac461fad97466ce55f1a39f99206d2dbd227e1db133dab1773c5eb220530" exitCode=0 Sep 09 23:18:53.399061 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:53.399009 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" event={"ID":"9eea67d0-b1b9-4305-811e-20aa6ea233ed","Type":"ContainerDied","Data":"b4f8ac461fad97466ce55f1a39f99206d2dbd227e1db133dab1773c5eb220530"} Sep 09 23:18:53.399300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:53.399280 2559 scope.go:117] "RemoveContainer" containerID="b4f8ac461fad97466ce55f1a39f99206d2dbd227e1db133dab1773c5eb220530" Sep 09 23:18:54.405014 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:54.404981 2559 generic.go:356] "Generic (PLEG): container finished" podID="7abba9dc-6fbe-4927-8298-ddda486d8ed1" containerID="4d364ea2a709e6abcf66a920629603f5675c6f47357ad8e6031ed1e2fd284282" exitCode=0 Sep 09 23:18:54.405384 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:54.405054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" event={"ID":"7abba9dc-6fbe-4927-8298-ddda486d8ed1","Type":"ContainerDied","Data":"4d364ea2a709e6abcf66a920629603f5675c6f47357ad8e6031ed1e2fd284282"} Sep 09 23:18:54.405445 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:54.405431 2559 scope.go:117] "RemoveContainer" containerID="4d364ea2a709e6abcf66a920629603f5675c6f47357ad8e6031ed1e2fd284282" Sep 09 23:18:54.407216 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:54.407191 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-sktw2" event={"ID":"9eea67d0-b1b9-4305-811e-20aa6ea233ed","Type":"ContainerStarted","Data":"cb8126c526ff688d4ea0c3827fd4b7aec4e1c03e92b7fd350940cc211bff5e1c"} Sep 09 23:18:55.412824 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:18:55.412783 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hrjpm" event={"ID":"7abba9dc-6fbe-4927-8298-ddda486d8ed1","Type":"ContainerStarted","Data":"915dc32a229fb321cd1cf6eb677c31735bad34a9c79c12f3fac8cfa9a0418ec3"} Sep 09 23:19:00.485679 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:00.485635 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" podUID="67d1f8cf-5d76-4d43-b63c-97933be1a863" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 23:19:10.485077 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:10.485033 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" podUID="67d1f8cf-5d76-4d43-b63c-97933be1a863" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 23:19:10.485736 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:10.485126 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" Sep 09 23:19:10.485830 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:10.485771 2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"0d90e9a66ec4ae26bac6a8b75ef0ced740f15437b7e3dfbafc5452f44ec06661"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 09 23:19:10.485830 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:10.485822 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" podUID="67d1f8cf-5d76-4d43-b63c-97933be1a863" containerName="service-proxy" containerID="cri-o://0d90e9a66ec4ae26bac6a8b75ef0ced740f15437b7e3dfbafc5452f44ec06661" gracePeriod=30 Sep 09 23:19:11.492203 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:11.492167 2559 generic.go:356] "Generic (PLEG): container finished" podID="67d1f8cf-5d76-4d43-b63c-97933be1a863" containerID="0d90e9a66ec4ae26bac6a8b75ef0ced740f15437b7e3dfbafc5452f44ec06661" exitCode=2 Sep 09 23:19:11.492203 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:11.492210 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" event={"ID":"67d1f8cf-5d76-4d43-b63c-97933be1a863","Type":"ContainerDied","Data":"0d90e9a66ec4ae26bac6a8b75ef0ced740f15437b7e3dfbafc5452f44ec06661"} Sep 09 23:19:11.492653 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:19:11.492242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6c9c9f4c49-v4wkz" event={"ID":"67d1f8cf-5d76-4d43-b63c-97933be1a863","Type":"ContainerStarted","Data":"75dfb8aef83d19eeae22c6d80137a39078f90265f613f45047210adc0ed21d69"} Sep 09 23:21:46.554125 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:21:46.554093 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:21:46.554941 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:21:46.554919 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:21:46.564715 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:21:46.564690 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:21:46.565202 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:21:46.565182 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:21:46.568157 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:21:46.568133 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 23:22:39.767468 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.767392 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg"] Sep 09 23:22:39.767876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.767689 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" containerName="registry" Sep 09 23:22:39.767876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.767700 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" containerName="registry" Sep 09 23:22:39.767876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.767715 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a67da6-2beb-4f01-bc0c-ae50d48783a1" containerName="console" Sep 09 23:22:39.767876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.767721 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a67da6-2beb-4f01-bc0c-ae50d48783a1" containerName="console" Sep 09 23:22:39.767876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.767781 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a67da6-2beb-4f01-bc0c-ae50d48783a1" containerName="console" Sep 09 23:22:39.767876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.767791 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e7b0c6-a294-485c-9e6e-aef7cd1f9ed8" containerName="registry" Sep 09 23:22:39.769640 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.769623 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:39.772415 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.772390 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 23:22:39.772415 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.772402 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 23:22:39.772681 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.772535 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 23:22:39.772681 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.772561 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 23:22:39.772760 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.772747 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7xcjn\"" Sep 09 23:22:39.806933 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.806905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg"] Sep 09 23:22:39.914308 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.914277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4mjn\" (UniqueName: \"kubernetes.io/projected/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-kube-api-access-k4mjn\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:39.914494 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.914316 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:39.914494 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.914341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-webhook-cert\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:39.914638 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.914482 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-manifests\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:39.914638 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:39.914516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-tmp\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.015345 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.015308 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-manifests\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.015345 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.015347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-tmp\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.015562 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.015388 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4mjn\" (UniqueName: \"kubernetes.io/projected/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-kube-api-access-k4mjn\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.015562 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.015412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.015562 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.015426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-webhook-cert\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.015832 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.015810 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-manifests\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.017821 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.017758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-tmp\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.018144 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.018123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-webhook-cert\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.018185 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.018123 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.022732 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.022711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4mjn\" (UniqueName: \"kubernetes.io/projected/b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f-kube-api-access-k4mjn\") pod \"opendatahub-operator-controller-manager-8c5c8967d-hptqg\" (UID: \"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f\") " pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.079910 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.079878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:40.354843 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.354817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg"] Sep 09 23:22:40.359566 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:22:40.359538 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a32b31_5731_4ea6_a9f4_e05eda6e6d5f.slice/crio-5bc9d56a9981148ce004d7e0a2c1b0a40827535c61d2973719f91fa8280d2540 WatchSource:0}: Error finding container 5bc9d56a9981148ce004d7e0a2c1b0a40827535c61d2973719f91fa8280d2540: Status 404 returned error can't find the container with id 5bc9d56a9981148ce004d7e0a2c1b0a40827535c61d2973719f91fa8280d2540 Sep 09 23:22:40.360973 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:40.360958 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:22:41.311170 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:41.311129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" event={"ID":"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f","Type":"ContainerStarted","Data":"5bc9d56a9981148ce004d7e0a2c1b0a40827535c61d2973719f91fa8280d2540"} Sep 09 23:22:43.321823 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:43.321792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" event={"ID":"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f","Type":"ContainerStarted","Data":"02620090f5583b8c80e6c122a5bb177e5a5149e805118094d4a7a71b5b54e83f"} Sep 09 23:22:44.326974 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:44.326949 2559 generic.go:356] "Generic (PLEG): container finished" podID="b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f" containerID="02620090f5583b8c80e6c122a5bb177e5a5149e805118094d4a7a71b5b54e83f" exitCode=0 Sep 09 23:22:44.327293 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:44.327013 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" event={"ID":"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f","Type":"ContainerDied","Data":"02620090f5583b8c80e6c122a5bb177e5a5149e805118094d4a7a71b5b54e83f"} Sep 09 23:22:45.333023 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:45.332986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" event={"ID":"b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f","Type":"ContainerStarted","Data":"7ebe324ec8c09d8530ca39520d81a29849b7249f84b0e1aedcd3b552b40dbeb3"} Sep 09 23:22:45.333445 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:45.333061 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:22:45.357679 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:45.357634 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" podStartSLOduration=2.194442836 podStartE2EDuration="6.357616938s" totalImagesPullingTime="3.149375751s" totalInitContainerRuntime="1.013798351s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:22:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:22:45.357132253 +0000 UTC m=+359.169740616" watchObservedRunningTime="2026-09-09 23:22:45.357616938 +0000 UTC m=+359.170225301" Sep 09 23:22:56.339957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:22:56.339924 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8c5c8967d-hptqg" Sep 09 23:23:17.289963 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.289926 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67998c845-lhwn7"] Sep 09 23:23:17.294394 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.294375 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.297294 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.297274 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 09 23:23:17.301354 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.301325 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 09 23:23:17.301465 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.301408 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 09 23:23:17.301465 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.301455 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4x7sb\"" Sep 09 23:23:17.302165 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.302141 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:23:17.302367 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.302349 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 09 23:23:17.314105 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.314081 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67998c845-lhwn7"] Sep 09 23:23:17.410300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.410265 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-metrics-cert\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.410502 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.410371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-manager-config\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.410502 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.410403 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9f8\" (UniqueName: \"kubernetes.io/projected/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-kube-api-access-6z9f8\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.410502 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.410441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-cert\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.511340 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.511312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-metrics-cert\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.511525 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.511399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-manager-config\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.511525 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.511429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9f8\" (UniqueName: \"kubernetes.io/projected/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-kube-api-access-6z9f8\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.511525 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.511471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-cert\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.512271 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.512237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-manager-config\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.514079 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.514056 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-cert\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.514176 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.514128 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-metrics-cert\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.519744 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.519724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9f8\" (UniqueName: \"kubernetes.io/projected/451a1f57-5b9f-4902-bd9f-e5a16a20fb59-kube-api-access-6z9f8\") pod \"lws-controller-manager-67998c845-lhwn7\" (UID: \"451a1f57-5b9f-4902-bd9f-e5a16a20fb59\") " pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.604888 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.604825 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:17.932475 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:17.932286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67998c845-lhwn7"] Sep 09 23:23:17.935970 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:23:17.935903 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451a1f57_5b9f_4902_bd9f_e5a16a20fb59.slice/crio-7dfa29091ce9ca604ec29be8826f67523c9a4aea896378aafbc3d3586d87a7dc WatchSource:0}: Error finding container 7dfa29091ce9ca604ec29be8826f67523c9a4aea896378aafbc3d3586d87a7dc: Status 404 returned error can't find the container with id 7dfa29091ce9ca604ec29be8826f67523c9a4aea896378aafbc3d3586d87a7dc Sep 09 23:23:18.480149 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.480112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" event={"ID":"451a1f57-5b9f-4902-bd9f-e5a16a20fb59","Type":"ContainerStarted","Data":"7dfa29091ce9ca604ec29be8826f67523c9a4aea896378aafbc3d3586d87a7dc"} Sep 09 23:23:18.764949 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.764850 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp"] Sep 09 23:23:18.773025 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.772195 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:18.775706 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.775678 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nrrks\"" Sep 09 23:23:18.775975 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.775956 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 23:23:18.784929 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.784607 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp"] Sep 09 23:23:18.828859 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.828824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:18.828985 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.828916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cnz\" (UniqueName: \"kubernetes.io/projected/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-kube-api-access-j6cnz\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:18.930718 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.930297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cnz\" (UniqueName: \"kubernetes.io/projected/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-kube-api-access-j6cnz\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:18.930718 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.930388 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:18.930718 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:23:18.930501 2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 09 23:23:18.930718 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:23:18.930567 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert podName:8369569d-4f02-4ef0-8cb8-fcc9e129b5a2 nodeName:}" failed. No retries permitted until 2026-09-09 23:23:19.430547355 +0000 UTC m=+393.243155695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert") pod "llmisvc-controller-manager-777c6bd7d5-qxxjp" (UID: "8369569d-4f02-4ef0-8cb8-fcc9e129b5a2") : secret "llmisvc-webhook-server-cert" not found Sep 09 23:23:18.940701 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:18.940671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cnz\" (UniqueName: \"kubernetes.io/projected/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-kube-api-access-j6cnz\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:19.437117 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:19.437080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:19.437300 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:23:19.437250 2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 09 23:23:19.437365 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:23:19.437326 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert podName:8369569d-4f02-4ef0-8cb8-fcc9e129b5a2 nodeName:}" failed. No retries permitted until 2026-09-09 23:23:20.437307369 +0000 UTC m=+394.249915711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert") pod "llmisvc-controller-manager-777c6bd7d5-qxxjp" (UID: "8369569d-4f02-4ef0-8cb8-fcc9e129b5a2") : secret "llmisvc-webhook-server-cert" not found Sep 09 23:23:20.447318 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:20.447278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:20.450137 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:20.450114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8369569d-4f02-4ef0-8cb8-fcc9e129b5a2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qxxjp\" (UID: \"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:20.592445 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:20.592422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:20.872311 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:20.872277 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp"] Sep 09 23:23:21.498609 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:21.498564 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" event={"ID":"451a1f57-5b9f-4902-bd9f-e5a16a20fb59","Type":"ContainerStarted","Data":"2ee466b5f1d791ddd092ae49bff97d4a8bb100293e766caeb9c5dbe00168ecc7"} Sep 09 23:23:21.498990 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:21.498645 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:21.499805 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:21.499779 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" event={"ID":"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2","Type":"ContainerStarted","Data":"bfb7fce85c2d6d8590b02d55169fec932daa5bad81c8e445008841c943309c31"} Sep 09 23:23:21.517756 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:21.517706 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" podStartSLOduration=1.886597184 podStartE2EDuration="4.517692212s" totalImagesPullingTime="2.631095028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:23:21.516151442 +0000 UTC m=+395.328759803" watchObservedRunningTime="2026-09-09 23:23:21.517692212 +0000 UTC m=+395.330300621" Sep 09 23:23:24.516304 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:24.516231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" event={"ID":"8369569d-4f02-4ef0-8cb8-fcc9e129b5a2","Type":"ContainerStarted","Data":"3c6d8b0791dd9e66c386b11d67c06d6804b1a2fc23ce93535e7584a37300f40b"} Sep 09 23:23:24.516670 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:24.516311 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:23:24.534716 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:24.534671 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" podStartSLOduration=3.800767257 podStartE2EDuration="6.534654443s" totalImagesPullingTime="2.733887186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:23:24.53210208 +0000 UTC m=+398.344710454" watchObservedRunningTime="2026-09-09 23:23:24.534654443 +0000 UTC m=+398.347262805" Sep 09 23:23:32.508628 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:32.508574 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67998c845-lhwn7" Sep 09 23:23:55.522959 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:23:55.522922 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qxxjp" Sep 09 23:24:30.570550 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.570509 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9"] Sep 09 23:24:30.575645 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.575619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:30.578247 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.578227 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 23:24:30.578375 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.578304 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 23:24:30.578440 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.578416 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n9mbb\"" Sep 09 23:24:30.583278 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.583258 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9"] Sep 09 23:24:30.682348 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.682319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/551f85ab-5625-4ba6-94cb-f85e86d2b901-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:30.682474 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.682388 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvxl\" (UniqueName: \"kubernetes.io/projected/551f85ab-5625-4ba6-94cb-f85e86d2b901-kube-api-access-vjvxl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:30.783204 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.783170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvxl\" (UniqueName: \"kubernetes.io/projected/551f85ab-5625-4ba6-94cb-f85e86d2b901-kube-api-access-vjvxl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:30.783329 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.783217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/551f85ab-5625-4ba6-94cb-f85e86d2b901-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:30.783553 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.783537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/551f85ab-5625-4ba6-94cb-f85e86d2b901-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:30.794325 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.794301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvxl\" (UniqueName: \"kubernetes.io/projected/551f85ab-5625-4ba6-94cb-f85e86d2b901-kube-api-access-vjvxl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:30.886517 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:30.886451 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:31.164411 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:31.164381 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9"] Sep 09 23:24:31.166413 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:24:31.166383 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551f85ab_5625_4ba6_94cb_f85e86d2b901.slice/crio-b939a474bedf330f059ee4cd349cc998d7b92f3f775a88b1b9208752a671a300 WatchSource:0}: Error finding container b939a474bedf330f059ee4cd349cc998d7b92f3f775a88b1b9208752a671a300: Status 404 returned error can't find the container with id b939a474bedf330f059ee4cd349cc998d7b92f3f775a88b1b9208752a671a300 Sep 09 23:24:31.815187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:31.815139 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" event={"ID":"551f85ab-5625-4ba6-94cb-f85e86d2b901","Type":"ContainerStarted","Data":"b939a474bedf330f059ee4cd349cc998d7b92f3f775a88b1b9208752a671a300"} Sep 09 23:24:35.843503 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:35.843467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" event={"ID":"551f85ab-5625-4ba6-94cb-f85e86d2b901","Type":"ContainerStarted","Data":"ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b"} Sep 09 23:24:35.843939 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:35.843579 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:35.870150 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:35.870102 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" podStartSLOduration=2.0928528220000002 podStartE2EDuration="5.870089144s" totalImagesPullingTime="3.777236322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:24:35.869880614 +0000 UTC m=+469.682488976" watchObservedRunningTime="2026-09-09 23:24:35.870089144 +0000 UTC m=+469.682697505" Sep 09 23:24:46.849701 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:46.849669 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:48.780795 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.780761 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s"] Sep 09 23:24:48.787001 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.786981 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:48.797637 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.797614 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s"] Sep 09 23:24:48.884077 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.884051 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s"] Sep 09 23:24:48.884263 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:48.884243 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-75s2q], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" podUID="37f79784-2699-4617-8c64-fa4da4bb2fc0" Sep 09 23:24:48.898727 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.898708 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:48.905832 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.905813 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:48.916839 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.916817 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74"] Sep 09 23:24:48.920505 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.920490 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:48.929268 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.929242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75s2q\" (UniqueName: \"kubernetes.io/projected/37f79784-2699-4617-8c64-fa4da4bb2fc0-kube-api-access-75s2q\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-z6j9s\" (UID: \"37f79784-2699-4617-8c64-fa4da4bb2fc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:48.929382 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.929310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37f79784-2699-4617-8c64-fa4da4bb2fc0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-z6j9s\" (UID: \"37f79784-2699-4617-8c64-fa4da4bb2fc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:48.934242 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.934222 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74"] Sep 09 23:24:48.934418 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:48.934401 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-9l9k2], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-9l9k2]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" podUID="551a9432-7d07-4635-a856-e060f06332f6" Sep 09 23:24:48.938160 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.938140 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74"] Sep 09 23:24:48.946329 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:48.946284 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"551a9432-7d07-4635-a856-e060f06332f6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-09T23:24:48Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-09T23:24:48Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-09T23:24:48Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-09-09T23:24:48Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/kuadrant\\\",\\\"name\\\":\\\"extensions-socket-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-9l9k2\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.142.167\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.142.167\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-09-09T23:24:48Z\\\"}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-5769f99d9f-mfs74\": pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:48.955533 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.955514 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9"] Sep 09 23:24:48.955710 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.955682 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" containerName="manager" containerID="cri-o://ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b" gracePeriod=2 Sep 09 23:24:48.958011 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.957988 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4"] Sep 09 23:24:48.958200 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:48.958160 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:48.961834 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.961806 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:48.970760 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.970736 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9"] Sep 09 23:24:48.971961 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:48.971930 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"551f85ab-5625-4ba6-94cb-f85e86d2b901\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\": pods \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:48.976001 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:48.975976 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:48.979118 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.979096 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4"] Sep 09 23:24:48.979479 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:48.979448 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:48.983182 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.983156 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv"] Sep 09 23:24:48.983555 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.983462 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" containerName="manager" Sep 09 23:24:48.983555 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.983475 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" containerName="manager" Sep 09 23:24:48.983555 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.983528 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" containerName="manager" Sep 09 23:24:48.986860 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.986841 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:48.991527 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:48.991510 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s"] Sep 09 23:24:49.000450 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.000428 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv"] Sep 09 23:24:49.002158 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.002135 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s"] Sep 09 23:24:49.009431 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.009393 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:49.029856 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.029830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvkv\" (UniqueName: \"kubernetes.io/projected/94501049-55f0-4319-9716-b1cb106304f2-kube-api-access-fbvkv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rp4t4\" (UID: \"94501049-55f0-4319-9716-b1cb106304f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.029971 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.029906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75s2q\" (UniqueName: \"kubernetes.io/projected/37f79784-2699-4617-8c64-fa4da4bb2fc0-kube-api-access-75s2q\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-z6j9s\" (UID: \"37f79784-2699-4617-8c64-fa4da4bb2fc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:49.029971 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.029962 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94501049-55f0-4319-9716-b1cb106304f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rp4t4\" (UID: \"94501049-55f0-4319-9716-b1cb106304f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.030085 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.030014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37f79784-2699-4617-8c64-fa4da4bb2fc0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-z6j9s\" (UID: \"37f79784-2699-4617-8c64-fa4da4bb2fc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:49.030465 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.030427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37f79784-2699-4617-8c64-fa4da4bb2fc0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-z6j9s\" (UID: \"37f79784-2699-4617-8c64-fa4da4bb2fc0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:49.036113 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.036078 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:49.036213 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.036193 2559 projected.go:196] Error preparing data for projected volume kube-api-access-75s2q for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s: failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" not found Sep 09 23:24:49.036280 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.036258 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f79784-2699-4617-8c64-fa4da4bb2fc0-kube-api-access-75s2q podName:37f79784-2699-4617-8c64-fa4da4bb2fc0 nodeName:}" failed. No retries permitted until 2026-09-09 23:24:49.536230262 +0000 UTC m=+483.348838602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-75s2q" (UniqueName: "kubernetes.io/projected/37f79784-2699-4617-8c64-fa4da4bb2fc0-kube-api-access-75s2q") pod "kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" (UID: "37f79784-2699-4617-8c64-fa4da4bb2fc0") : failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" not found Sep 09 23:24:49.131138 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131106 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37f79784-2699-4617-8c64-fa4da4bb2fc0-extensions-socket-volume\") pod \"37f79784-2699-4617-8c64-fa4da4bb2fc0\" (UID: \"37f79784-2699-4617-8c64-fa4da4bb2fc0\") " Sep 09 23:24:49.131396 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvkv\" (UniqueName: \"kubernetes.io/projected/94501049-55f0-4319-9716-b1cb106304f2-kube-api-access-fbvkv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rp4t4\" (UID: \"94501049-55f0-4319-9716-b1cb106304f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.131517 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131394 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f79784-2699-4617-8c64-fa4da4bb2fc0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "37f79784-2699-4617-8c64-fa4da4bb2fc0" (UID: "37f79784-2699-4617-8c64-fa4da4bb2fc0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:24:49.131517 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvw6\" (UniqueName: \"kubernetes.io/projected/f5fc0973-a276-4631-92bf-1e535631de21-kube-api-access-6qvw6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xrnkv\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.131517 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5fc0973-a276-4631-92bf-1e535631de21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xrnkv\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.131517 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94501049-55f0-4319-9716-b1cb106304f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rp4t4\" (UID: \"94501049-55f0-4319-9716-b1cb106304f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.131756 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131553 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75s2q\" (UniqueName: \"kubernetes.io/projected/37f79784-2699-4617-8c64-fa4da4bb2fc0-kube-api-access-75s2q\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:24:49.131756 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131571 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37f79784-2699-4617-8c64-fa4da4bb2fc0-extensions-socket-volume\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:24:49.132004 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.131983 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94501049-55f0-4319-9716-b1cb106304f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rp4t4\" (UID: \"94501049-55f0-4319-9716-b1cb106304f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.142563 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.142535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvkv\" (UniqueName: \"kubernetes.io/projected/94501049-55f0-4319-9716-b1cb106304f2-kube-api-access-fbvkv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rp4t4\" (UID: \"94501049-55f0-4319-9716-b1cb106304f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.193583 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.193562 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:49.196168 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.196142 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:49.198269 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.198247 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:49.232677 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.232648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvw6\" (UniqueName: \"kubernetes.io/projected/f5fc0973-a276-4631-92bf-1e535631de21-kube-api-access-6qvw6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xrnkv\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.232765 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.232687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5fc0973-a276-4631-92bf-1e535631de21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xrnkv\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.233092 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.233073 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5fc0973-a276-4631-92bf-1e535631de21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xrnkv\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.247573 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.247549 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvw6\" (UniqueName: \"kubernetes.io/projected/f5fc0973-a276-4631-92bf-1e535631de21-kube-api-access-6qvw6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xrnkv\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.328553 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.328488 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.333109 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.333090 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjvxl\" (UniqueName: \"kubernetes.io/projected/551f85ab-5625-4ba6-94cb-f85e86d2b901-kube-api-access-vjvxl\") pod \"551f85ab-5625-4ba6-94cb-f85e86d2b901\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " Sep 09 23:24:49.333186 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.333162 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/551f85ab-5625-4ba6-94cb-f85e86d2b901-extensions-socket-volume\") pod \"551f85ab-5625-4ba6-94cb-f85e86d2b901\" (UID: \"551f85ab-5625-4ba6-94cb-f85e86d2b901\") " Sep 09 23:24:49.333540 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.333519 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551f85ab-5625-4ba6-94cb-f85e86d2b901-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "551f85ab-5625-4ba6-94cb-f85e86d2b901" (UID: "551f85ab-5625-4ba6-94cb-f85e86d2b901"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:24:49.335281 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.335264 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.335356 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.335262 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551f85ab-5625-4ba6-94cb-f85e86d2b901-kube-api-access-vjvxl" (OuterVolumeSpecName: "kube-api-access-vjvxl") pod "551f85ab-5625-4ba6-94cb-f85e86d2b901" (UID: "551f85ab-5625-4ba6-94cb-f85e86d2b901"). InnerVolumeSpecName "kube-api-access-vjvxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:24:49.433724 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.433699 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/551f85ab-5625-4ba6-94cb-f85e86d2b901-extensions-socket-volume\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:24:49.433724 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.433724 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvxl\" (UniqueName: \"kubernetes.io/projected/551f85ab-5625-4ba6-94cb-f85e86d2b901-kube-api-access-vjvxl\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:24:49.617613 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.617557 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4"] Sep 09 23:24:49.618850 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:24:49.618818 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fc0973_a276_4631_92bf_1e535631de21.slice/crio-5491e5af3e030b2a739e196471cf86cd99209c94b7ba5a3cb413c106b6ca0161 WatchSource:0}: Error finding container 5491e5af3e030b2a739e196471cf86cd99209c94b7ba5a3cb413c106b6ca0161: Status 404 returned error can't find the container with id 5491e5af3e030b2a739e196471cf86cd99209c94b7ba5a3cb413c106b6ca0161 Sep 09 23:24:49.619045 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.619024 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv"] Sep 09 23:24:49.619530 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:24:49.619516 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94501049_55f0_4319_9716_b1cb106304f2.slice/crio-5d636346050b7c5790ac80e31173c9081a47f0dc730a7054309062b964b2add7 WatchSource:0}: Error finding container 5d636346050b7c5790ac80e31173c9081a47f0dc730a7054309062b964b2add7: Status 404 returned error can't find the container with id 5d636346050b7c5790ac80e31173c9081a47f0dc730a7054309062b964b2add7 Sep 09 23:24:49.906434 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.906354 2559 generic.go:356] "Generic (PLEG): container finished" podID="551f85ab-5625-4ba6-94cb-f85e86d2b901" containerID="ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b" exitCode=0 Sep 09 23:24:49.906434 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.906400 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:49.906949 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.906456 2559 scope.go:117] "RemoveContainer" containerID="ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b" Sep 09 23:24:49.908462 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.908436 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" event={"ID":"f5fc0973-a276-4631-92bf-1e535631de21","Type":"ContainerStarted","Data":"a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc"} Sep 09 23:24:49.908607 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.908500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" event={"ID":"f5fc0973-a276-4631-92bf-1e535631de21","Type":"ContainerStarted","Data":"5491e5af3e030b2a739e196471cf86cd99209c94b7ba5a3cb413c106b6ca0161"} Sep 09 23:24:49.908607 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.908550 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:24:49.909102 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.909009 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:49.910544 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.910519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" event={"ID":"94501049-55f0-4319-9716-b1cb106304f2","Type":"ContainerStarted","Data":"76a9480c7e4a8114b6cdd7976b421c8f505f8966c15cca5e3bbf546f300bafa8"} Sep 09 23:24:49.910693 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.910554 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:49.910693 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.910566 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" event={"ID":"94501049-55f0-4319-9716-b1cb106304f2","Type":"ContainerStarted","Data":"5d636346050b7c5790ac80e31173c9081a47f0dc730a7054309062b964b2add7"} Sep 09 23:24:49.910824 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.910760 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:24:49.910824 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.910804 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:49.912055 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.912028 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:49.914257 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.914229 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-z6j9s\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="37f79784-2699-4617-8c64-fa4da4bb2fc0" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:49.916704 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.916685 2559 scope.go:117] "RemoveContainer" containerID="ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b" Sep 09 23:24:49.917002 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.916976 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b\": container with ID starting with ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b not found: ID does not exist" containerID="ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b" Sep 09 23:24:49.917081 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.917011 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b"} err="failed to get container status \"ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b\": rpc error: code = NotFound desc = could not find container \"ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b\": container with ID starting with ca9bb547a242b19a730bc8c1110224ef975dafc8a588df89b558c5aa8919160b not found: ID does not exist" Sep 09 23:24:49.921436 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.921420 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:49.950478 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.950424 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" podStartSLOduration=1.9504040360000001 podStartE2EDuration="1.950404036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:24:49.946184102 +0000 UTC m=+483.758792485" watchObservedRunningTime="2026-09-09 23:24:49.950404036 +0000 UTC m=+483.763012398" Sep 09 23:24:49.953006 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.952951 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:49.956134 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.956108 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:49.992898 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:49.992848 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" podStartSLOduration=1.992832581 podStartE2EDuration="1.992832581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:24:49.99060292 +0000 UTC m=+483.803211274" watchObservedRunningTime="2026-09-09 23:24:49.992832581 +0000 UTC m=+483.805440943" Sep 09 23:24:49.993105 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.993080 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lf4g9\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lf4g9" Sep 09 23:24:49.996171 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.996146 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-z6j9s\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="37f79784-2699-4617-8c64-fa4da4bb2fc0" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-z6j9s" Sep 09 23:24:49.998309 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:49.998288 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:50.644828 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:50.644794 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f79784-2699-4617-8c64-fa4da4bb2fc0" path="/var/lib/kubelet/pods/37f79784-2699-4617-8c64-fa4da4bb2fc0/volumes" Sep 09 23:24:50.644983 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:50.644966 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="551a9432-7d07-4635-a856-e060f06332f6" path="/var/lib/kubelet/pods/551a9432-7d07-4635-a856-e060f06332f6/volumes" Sep 09 23:24:50.645171 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:50.645158 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="551f85ab-5625-4ba6-94cb-f85e86d2b901" path="/var/lib/kubelet/pods/551f85ab-5625-4ba6-94cb-f85e86d2b901/volumes" Sep 09 23:24:50.916323 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:24:50.916240 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:50.918902 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:50.918870 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:24:50.921050 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:24:50.921022 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-mfs74\" is forbidden: User \"system:node:ip-10-0-142-167.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-167.ec2.internal' and this object" podUID="551a9432-7d07-4635-a856-e060f06332f6" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mfs74" Sep 09 23:25:00.918458 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:00.918426 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rp4t4" Sep 09 23:25:00.918891 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:00.918492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:25:00.993648 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:00.993612 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv"] Sep 09 23:25:00.993810 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:00.993790 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" podUID="f5fc0973-a276-4631-92bf-1e535631de21" containerName="manager" containerID="cri-o://a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc" gracePeriod=10 Sep 09 23:25:01.242747 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.242721 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:25:01.434243 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.434215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qvw6\" (UniqueName: \"kubernetes.io/projected/f5fc0973-a276-4631-92bf-1e535631de21-kube-api-access-6qvw6\") pod \"f5fc0973-a276-4631-92bf-1e535631de21\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " Sep 09 23:25:01.434445 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.434292 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5fc0973-a276-4631-92bf-1e535631de21-extensions-socket-volume\") pod \"f5fc0973-a276-4631-92bf-1e535631de21\" (UID: \"f5fc0973-a276-4631-92bf-1e535631de21\") " Sep 09 23:25:01.434658 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.434636 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fc0973-a276-4631-92bf-1e535631de21-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f5fc0973-a276-4631-92bf-1e535631de21" (UID: "f5fc0973-a276-4631-92bf-1e535631de21"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:25:01.436173 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.436154 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc0973-a276-4631-92bf-1e535631de21-kube-api-access-6qvw6" (OuterVolumeSpecName: "kube-api-access-6qvw6") pod "f5fc0973-a276-4631-92bf-1e535631de21" (UID: "f5fc0973-a276-4631-92bf-1e535631de21"). InnerVolumeSpecName "kube-api-access-6qvw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:25:01.534971 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.534946 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5fc0973-a276-4631-92bf-1e535631de21-extensions-socket-volume\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:25:01.534971 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.534969 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvw6\" (UniqueName: \"kubernetes.io/projected/f5fc0973-a276-4631-92bf-1e535631de21-kube-api-access-6qvw6\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:25:01.970194 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.970157 2559 generic.go:356] "Generic (PLEG): container finished" podID="f5fc0973-a276-4631-92bf-1e535631de21" containerID="a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc" exitCode=0 Sep 09 23:25:01.970579 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.970202 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" Sep 09 23:25:01.970579 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.970237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" event={"ID":"f5fc0973-a276-4631-92bf-1e535631de21","Type":"ContainerDied","Data":"a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc"} Sep 09 23:25:01.970579 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.970298 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv" event={"ID":"f5fc0973-a276-4631-92bf-1e535631de21","Type":"ContainerDied","Data":"5491e5af3e030b2a739e196471cf86cd99209c94b7ba5a3cb413c106b6ca0161"} Sep 09 23:25:01.970579 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.970324 2559 scope.go:117] "RemoveContainer" containerID="a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc" Sep 09 23:25:01.978932 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.978917 2559 scope.go:117] "RemoveContainer" containerID="a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc" Sep 09 23:25:01.979196 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:25:01.979180 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc\": container with ID starting with a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc not found: ID does not exist" containerID="a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc" Sep 09 23:25:01.979237 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.979203 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc"} err="failed to get container status \"a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc\": rpc error: code = NotFound desc = could not find container \"a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc\": container with ID starting with a2786eaaea7038575260f2cda5d38d34ecaf718597673d63d80137e5ad0ec5fc not found: ID does not exist" Sep 09 23:25:01.995206 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.995183 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv"] Sep 09 23:25:01.998816 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:01.998793 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xrnkv"] Sep 09 23:25:02.643312 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:02.643271 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5fc0973-a276-4631-92bf-1e535631de21" path="/var/lib/kubelet/pods/f5fc0973-a276-4631-92bf-1e535631de21/volumes" Sep 09 23:25:19.366852 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.366818 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fj96k"] Sep 09 23:25:19.367247 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.367145 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fc0973-a276-4631-92bf-1e535631de21" containerName="manager" Sep 09 23:25:19.367247 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.367156 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fc0973-a276-4631-92bf-1e535631de21" containerName="manager" Sep 09 23:25:19.367247 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.367219 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fc0973-a276-4631-92bf-1e535631de21" containerName="manager" Sep 09 23:25:19.370225 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.370207 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.372881 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.372857 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 23:25:19.372881 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.372871 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7nt94\"" Sep 09 23:25:19.379406 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.379383 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fj96k"] Sep 09 23:25:19.457242 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.457204 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fj96k"] Sep 09 23:25:19.476329 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.476297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-config-file\") pod \"limitador-limitador-8588749977-fj96k\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.476488 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.476350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfvb\" (UniqueName: \"kubernetes.io/projected/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-kube-api-access-zkfvb\") pod \"limitador-limitador-8588749977-fj96k\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.577833 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.577802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-config-file\") pod \"limitador-limitador-8588749977-fj96k\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.577991 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.577867 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfvb\" (UniqueName: \"kubernetes.io/projected/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-kube-api-access-zkfvb\") pod \"limitador-limitador-8588749977-fj96k\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.578518 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.578487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-config-file\") pod \"limitador-limitador-8588749977-fj96k\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.586094 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.586064 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfvb\" (UniqueName: \"kubernetes.io/projected/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-kube-api-access-zkfvb\") pod \"limitador-limitador-8588749977-fj96k\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.681458 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.681428 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:19.949253 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:19.949225 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fj96k"] Sep 09 23:25:19.953844 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:25:19.952244 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79553f4b_c6a5_4b49_a5e5_6df424c8fcf0.slice/crio-1a3faff26e1a19758accc94f3dea9ab9ca3a6ebcb29f0a38d472b01e9784c270 WatchSource:0}: Error finding container 1a3faff26e1a19758accc94f3dea9ab9ca3a6ebcb29f0a38d472b01e9784c270: Status 404 returned error can't find the container with id 1a3faff26e1a19758accc94f3dea9ab9ca3a6ebcb29f0a38d472b01e9784c270 Sep 09 23:25:20.054522 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:20.054487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" event={"ID":"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0","Type":"ContainerStarted","Data":"1a3faff26e1a19758accc94f3dea9ab9ca3a6ebcb29f0a38d472b01e9784c270"} Sep 09 23:25:22.066430 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:22.066392 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" event={"ID":"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0","Type":"ContainerStarted","Data":"9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b"} Sep 09 23:25:22.066902 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:22.066518 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:22.084401 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:22.084355 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" podStartSLOduration=1.980097827 podStartE2EDuration="3.084340984s" totalImagesPullingTime="1.104243157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:25:22.082886005 +0000 UTC m=+515.895494367" watchObservedRunningTime="2026-09-09 23:25:22.084340984 +0000 UTC m=+515.896949345" Sep 09 23:25:32.472570 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:32.472529 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fj96k"] Sep 09 23:25:32.473085 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:32.472762 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" podUID="79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" containerName="limitador" containerID="cri-o://9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b" gracePeriod=30 Sep 09 23:25:32.473667 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:32.473442 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:33.018853 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.018829 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:33.086797 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.086736 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-config-file\") pod \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " Sep 09 23:25:33.086797 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.086793 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkfvb\" (UniqueName: \"kubernetes.io/projected/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-kube-api-access-zkfvb\") pod \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\" (UID: \"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0\") " Sep 09 23:25:33.087131 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.087101 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-config-file" (OuterVolumeSpecName: "config-file") pod "79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" (UID: "79553f4b-c6a5-4b49-a5e5-6df424c8fcf0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:25:33.088813 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.088785 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-kube-api-access-zkfvb" (OuterVolumeSpecName: "kube-api-access-zkfvb") pod "79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" (UID: "79553f4b-c6a5-4b49-a5e5-6df424c8fcf0"). InnerVolumeSpecName "kube-api-access-zkfvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:25:33.117995 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.117969 2559 generic.go:356] "Generic (PLEG): container finished" podID="79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" containerID="9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b" exitCode=0 Sep 09 23:25:33.118101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.118013 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" event={"ID":"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0","Type":"ContainerDied","Data":"9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b"} Sep 09 23:25:33.118101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.118021 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" Sep 09 23:25:33.118101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.118053 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-fj96k" event={"ID":"79553f4b-c6a5-4b49-a5e5-6df424c8fcf0","Type":"ContainerDied","Data":"1a3faff26e1a19758accc94f3dea9ab9ca3a6ebcb29f0a38d472b01e9784c270"} Sep 09 23:25:33.118101 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.118074 2559 scope.go:117] "RemoveContainer" containerID="9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b" Sep 09 23:25:33.127391 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.127367 2559 scope.go:117] "RemoveContainer" containerID="9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b" Sep 09 23:25:33.131543 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:25:33.131503 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b\": container with ID starting with 9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b not found: ID does not exist" containerID="9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b" Sep 09 23:25:33.131649 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.131553 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b"} err="failed to get container status \"9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b\": rpc error: code = NotFound desc = could not find container \"9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b\": container with ID starting with 9d1ff7dfc3cc176e06b6a346a466fceb57b08ba39f9107e8716a90ab5279cd7b not found: ID does not exist" Sep 09 23:25:33.144951 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.144929 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fj96k"] Sep 09 23:25:33.148106 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.148085 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-fj96k"] Sep 09 23:25:33.187883 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.187860 2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-config-file\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:25:33.187883 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:33.187880 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfvb\" (UniqueName: \"kubernetes.io/projected/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0-kube-api-access-zkfvb\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:25:34.643841 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:34.643801 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" path="/var/lib/kubelet/pods/79553f4b-c6a5-4b49-a5e5-6df424c8fcf0/volumes" Sep 09 23:25:35.880769 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.880734 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq"] Sep 09 23:25:35.881115 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.881074 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" containerName="limitador" Sep 09 23:25:35.881115 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.881088 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" containerName="limitador" Sep 09 23:25:35.881184 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.881156 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="79553f4b-c6a5-4b49-a5e5-6df424c8fcf0" containerName="limitador" Sep 09 23:25:35.885605 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.885573 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:35.887957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.887936 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 23:25:35.888132 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.888114 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 09 23:25:35.888195 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.888132 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 23:25:35.889323 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.889303 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nvx7d\"" Sep 09 23:25:35.891388 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:35.891366 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq"] Sep 09 23:25:36.010778 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.010750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cbaeb6d0-6212-4d14-a561-3990ec5befdd-data\") pod \"postgres-69fb4594cf-lt9mq\" (UID: \"cbaeb6d0-6212-4d14-a561-3990ec5befdd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:36.010778 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.010779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5tj7\" (UniqueName: \"kubernetes.io/projected/cbaeb6d0-6212-4d14-a561-3990ec5befdd-kube-api-access-x5tj7\") pod \"postgres-69fb4594cf-lt9mq\" (UID: \"cbaeb6d0-6212-4d14-a561-3990ec5befdd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:36.111134 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.111109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cbaeb6d0-6212-4d14-a561-3990ec5befdd-data\") pod \"postgres-69fb4594cf-lt9mq\" (UID: \"cbaeb6d0-6212-4d14-a561-3990ec5befdd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:36.111234 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.111138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5tj7\" (UniqueName: \"kubernetes.io/projected/cbaeb6d0-6212-4d14-a561-3990ec5befdd-kube-api-access-x5tj7\") pod \"postgres-69fb4594cf-lt9mq\" (UID: \"cbaeb6d0-6212-4d14-a561-3990ec5befdd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:36.111458 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.111440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cbaeb6d0-6212-4d14-a561-3990ec5befdd-data\") pod \"postgres-69fb4594cf-lt9mq\" (UID: \"cbaeb6d0-6212-4d14-a561-3990ec5befdd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:36.120950 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.120931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5tj7\" (UniqueName: \"kubernetes.io/projected/cbaeb6d0-6212-4d14-a561-3990ec5befdd-kube-api-access-x5tj7\") pod \"postgres-69fb4594cf-lt9mq\" (UID: \"cbaeb6d0-6212-4d14-a561-3990ec5befdd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:36.196916 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.196894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:36.464218 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:36.464196 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq"] Sep 09 23:25:36.466255 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:25:36.466233 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaeb6d0_6212_4d14_a561_3990ec5befdd.slice/crio-f00226e6b97ef0272f9800f84625198971315983c8d17bd4ef5682038a0cf385 WatchSource:0}: Error finding container f00226e6b97ef0272f9800f84625198971315983c8d17bd4ef5682038a0cf385: Status 404 returned error can't find the container with id f00226e6b97ef0272f9800f84625198971315983c8d17bd4ef5682038a0cf385 Sep 09 23:25:37.137919 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:37.137885 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" event={"ID":"cbaeb6d0-6212-4d14-a561-3990ec5befdd","Type":"ContainerStarted","Data":"f00226e6b97ef0272f9800f84625198971315983c8d17bd4ef5682038a0cf385"} Sep 09 23:25:42.165087 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:42.165052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" event={"ID":"cbaeb6d0-6212-4d14-a561-3990ec5befdd","Type":"ContainerStarted","Data":"5bdb4d9bab31bd97a9a8446f4340d5c242aa2e9de4710d9207947f6367cf165c"} Sep 09 23:25:42.165523 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:42.165146 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:42.181609 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:42.181548 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" podStartSLOduration=2.380537095 podStartE2EDuration="7.181535941s" totalImagesPullingTime="4.800998846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:25:42.179936618 +0000 UTC m=+535.992544981" watchObservedRunningTime="2026-09-09 23:25:42.181535941 +0000 UTC m=+535.994144304" Sep 09 23:25:48.197454 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:48.197422 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lt9mq" Sep 09 23:25:55.305710 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.305675 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58cd95d57-bg7pr"] Sep 09 23:25:55.329415 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.329377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58cd95d57-bg7pr"] Sep 09 23:25:55.329571 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.329523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.332285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.332228 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 23:25:55.332285 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.332229 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 23:25:55.332466 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.332370 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 23:25:55.332466 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.332429 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cdf5c\"" Sep 09 23:25:55.468981 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.468949 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9bdb9bd9b-qrcx8"] Sep 09 23:25:55.470725 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.470702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-metrics-certs\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.470812 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.470761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjsqb\" (UniqueName: \"kubernetes.io/projected/f271d574-201a-4900-a910-00bb0fb5d3b2-kube-api-access-bjsqb\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.470866 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.470832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.472817 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.472798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.481474 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.481451 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9bdb9bd9b-qrcx8"] Sep 09 23:25:55.571929 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.571869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjsqb\" (UniqueName: \"kubernetes.io/projected/f271d574-201a-4900-a910-00bb0fb5d3b2-kube-api-access-bjsqb\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.571929 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.571906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077c5ec5-8276-4918-9168-8ffd8cf9531c-metrics-certs\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.572057 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.572033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/077c5ec5-8276-4918-9168-8ffd8cf9531c-webhook-cert\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.572092 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.572073 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.572131 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.572119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-metrics-certs\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.572173 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.572144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bn9\" (UniqueName: \"kubernetes.io/projected/077c5ec5-8276-4918-9168-8ffd8cf9531c-kube-api-access-68bn9\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.572269 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:25:55.572248 2559 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 09 23:25:55.572346 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:25:55.572335 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert podName:f271d574-201a-4900-a910-00bb0fb5d3b2 nodeName:}" failed. No retries permitted until 2026-09-09 23:25:56.072313419 +0000 UTC m=+549.884921760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert") pod "maas-controller-58cd95d57-bg7pr" (UID: "f271d574-201a-4900-a910-00bb0fb5d3b2") : secret "maas-controller-webhook-cert" not found Sep 09 23:25:55.574606 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.574569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-metrics-certs\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.581858 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.581838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjsqb\" (UniqueName: \"kubernetes.io/projected/f271d574-201a-4900-a910-00bb0fb5d3b2-kube-api-access-bjsqb\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:55.672798 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.672766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/077c5ec5-8276-4918-9168-8ffd8cf9531c-webhook-cert\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.672930 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.672854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bn9\" (UniqueName: \"kubernetes.io/projected/077c5ec5-8276-4918-9168-8ffd8cf9531c-kube-api-access-68bn9\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.672930 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.672916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077c5ec5-8276-4918-9168-8ffd8cf9531c-metrics-certs\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.675029 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.675009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/077c5ec5-8276-4918-9168-8ffd8cf9531c-webhook-cert\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.675115 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.675052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/077c5ec5-8276-4918-9168-8ffd8cf9531c-metrics-certs\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.679832 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.679815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bn9\" (UniqueName: \"kubernetes.io/projected/077c5ec5-8276-4918-9168-8ffd8cf9531c-kube-api-access-68bn9\") pod \"maas-controller-9bdb9bd9b-qrcx8\" (UID: \"077c5ec5-8276-4918-9168-8ffd8cf9531c\") " pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:55.783029 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:55.782999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:56.051812 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:56.051787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9bdb9bd9b-qrcx8"] Sep 09 23:25:56.054077 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:25:56.054045 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077c5ec5_8276_4918_9168_8ffd8cf9531c.slice/crio-34f6ee7c51641454b220f20b9ce0e6a0d7b530793c8ff8c743abcc485a49e6fe WatchSource:0}: Error finding container 34f6ee7c51641454b220f20b9ce0e6a0d7b530793c8ff8c743abcc485a49e6fe: Status 404 returned error can't find the container with id 34f6ee7c51641454b220f20b9ce0e6a0d7b530793c8ff8c743abcc485a49e6fe Sep 09 23:25:56.076800 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:56.076775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:56.078964 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:56.078923 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert\") pod \"maas-controller-58cd95d57-bg7pr\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:56.228854 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:56.228825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" event={"ID":"077c5ec5-8276-4918-9168-8ffd8cf9531c","Type":"ContainerStarted","Data":"34f6ee7c51641454b220f20b9ce0e6a0d7b530793c8ff8c743abcc485a49e6fe"} Sep 09 23:25:56.243969 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:56.243945 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:56.510517 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:56.510490 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58cd95d57-bg7pr"] Sep 09 23:25:56.512992 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:25:56.512965 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf271d574_201a_4900_a910_00bb0fb5d3b2.slice/crio-5ae96d0af369eafd1a6bb0b883ad08566810c2c64786fd458c1a37fea90d56a1 WatchSource:0}: Error finding container 5ae96d0af369eafd1a6bb0b883ad08566810c2c64786fd458c1a37fea90d56a1: Status 404 returned error can't find the container with id 5ae96d0af369eafd1a6bb0b883ad08566810c2c64786fd458c1a37fea90d56a1 Sep 09 23:25:57.238206 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:57.238170 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58cd95d57-bg7pr" event={"ID":"f271d574-201a-4900-a910-00bb0fb5d3b2","Type":"ContainerStarted","Data":"5ae96d0af369eafd1a6bb0b883ad08566810c2c64786fd458c1a37fea90d56a1"} Sep 09 23:25:59.250520 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:59.250362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58cd95d57-bg7pr" event={"ID":"f271d574-201a-4900-a910-00bb0fb5d3b2","Type":"ContainerStarted","Data":"519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0"} Sep 09 23:25:59.250520 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:59.250484 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:25:59.252979 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:59.252937 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" event={"ID":"077c5ec5-8276-4918-9168-8ffd8cf9531c","Type":"ContainerStarted","Data":"9f42ce053557e7499a25ce03100e03e5fd919f83e4d80c84527676f8b7c017bd"} Sep 09 23:25:59.253126 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:59.253084 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:25:59.267788 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:59.267740 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58cd95d57-bg7pr" podStartSLOduration=2.194914088 podStartE2EDuration="4.26772491s" totalImagesPullingTime="2.072810822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:25:59.266733254 +0000 UTC m=+553.079341642" watchObservedRunningTime="2026-09-09 23:25:59.26772491 +0000 UTC m=+553.080333271" Sep 09 23:25:59.284630 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:25:59.284562 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" podStartSLOduration=1.758993951 podStartE2EDuration="4.284546781s" totalImagesPullingTime="2.52555283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:25:59.284030399 +0000 UTC m=+553.096638763" watchObservedRunningTime="2026-09-09 23:25:59.284546781 +0000 UTC m=+553.097155121" Sep 09 23:26:10.259876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.259840 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9bdb9bd9b-qrcx8" Sep 09 23:26:10.260251 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.259928 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:26:10.322831 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.322795 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58cd95d57-bg7pr"] Sep 09 23:26:10.323027 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.323001 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58cd95d57-bg7pr" podUID="f271d574-201a-4900-a910-00bb0fb5d3b2" containerName="manager" containerID="cri-o://519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0" gracePeriod=30 Sep 09 23:26:10.575277 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.575253 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:26:10.703512 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.703479 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjsqb\" (UniqueName: \"kubernetes.io/projected/f271d574-201a-4900-a910-00bb0fb5d3b2-kube-api-access-bjsqb\") pod \"f271d574-201a-4900-a910-00bb0fb5d3b2\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " Sep 09 23:26:10.703512 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.703519 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert\") pod \"f271d574-201a-4900-a910-00bb0fb5d3b2\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " Sep 09 23:26:10.703751 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.703630 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-metrics-certs\") pod \"f271d574-201a-4900-a910-00bb0fb5d3b2\" (UID: \"f271d574-201a-4900-a910-00bb0fb5d3b2\") " Sep 09 23:26:10.705769 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.705733 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f271d574-201a-4900-a910-00bb0fb5d3b2" (UID: "f271d574-201a-4900-a910-00bb0fb5d3b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:26:10.705876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.705778 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f271d574-201a-4900-a910-00bb0fb5d3b2" (UID: "f271d574-201a-4900-a910-00bb0fb5d3b2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:26:10.705876 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.705834 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f271d574-201a-4900-a910-00bb0fb5d3b2-kube-api-access-bjsqb" (OuterVolumeSpecName: "kube-api-access-bjsqb") pod "f271d574-201a-4900-a910-00bb0fb5d3b2" (UID: "f271d574-201a-4900-a910-00bb0fb5d3b2"). InnerVolumeSpecName "kube-api-access-bjsqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:26:10.804896 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.804861 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-metrics-certs\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:26:10.804896 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.804894 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjsqb\" (UniqueName: \"kubernetes.io/projected/f271d574-201a-4900-a910-00bb0fb5d3b2-kube-api-access-bjsqb\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:26:10.805079 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:10.804909 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f271d574-201a-4900-a910-00bb0fb5d3b2-webhook-cert\") on node \"ip-10-0-142-167.ec2.internal\" DevicePath \"\"" Sep 09 23:26:11.310957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.310928 2559 generic.go:356] "Generic (PLEG): container finished" podID="f271d574-201a-4900-a910-00bb0fb5d3b2" containerID="519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0" exitCode=0 Sep 09 23:26:11.311333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.310984 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58cd95d57-bg7pr" Sep 09 23:26:11.311333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.310994 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58cd95d57-bg7pr" event={"ID":"f271d574-201a-4900-a910-00bb0fb5d3b2","Type":"ContainerDied","Data":"519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0"} Sep 09 23:26:11.311333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.311034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58cd95d57-bg7pr" event={"ID":"f271d574-201a-4900-a910-00bb0fb5d3b2","Type":"ContainerDied","Data":"5ae96d0af369eafd1a6bb0b883ad08566810c2c64786fd458c1a37fea90d56a1"} Sep 09 23:26:11.311333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.311056 2559 scope.go:117] "RemoveContainer" containerID="519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0" Sep 09 23:26:11.320102 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.320082 2559 scope.go:117] "RemoveContainer" containerID="519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0" Sep 09 23:26:11.320354 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:11.320334 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0\": container with ID starting with 519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0 not found: ID does not exist" containerID="519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0" Sep 09 23:26:11.320416 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.320362 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0"} err="failed to get container status \"519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0\": rpc error: code = NotFound desc = could not find container \"519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0\": container with ID starting with 519732bc0a44bfb998d8caa065ecc0b47fd7f89ad142275b4b1f6ef7eea11eb0 not found: ID does not exist" Sep 09 23:26:11.331193 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.331170 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58cd95d57-bg7pr"] Sep 09 23:26:11.334718 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:11.334697 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58cd95d57-bg7pr"] Sep 09 23:26:12.644006 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:12.643968 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f271d574-201a-4900-a910-00bb0fb5d3b2" path="/var/lib/kubelet/pods/f271d574-201a-4900-a910-00bb0fb5d3b2/volumes" Sep 09 23:26:31.962114 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.962080 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5"] Sep 09 23:26:31.962533 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.962449 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f271d574-201a-4900-a910-00bb0fb5d3b2" containerName="manager" Sep 09 23:26:31.962533 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.962462 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f271d574-201a-4900-a910-00bb0fb5d3b2" containerName="manager" Sep 09 23:26:31.962533 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.962525 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f271d574-201a-4900-a910-00bb0fb5d3b2" containerName="manager" Sep 09 23:26:31.968679 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.968656 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:31.971691 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.971669 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 23:26:31.972912 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.972891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 23:26:31.973004 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.972895 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d5cmg\"" Sep 09 23:26:31.975957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:31.975869 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5"] Sep 09 23:26:32.064090 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.064062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzn4v\" (UniqueName: \"kubernetes.io/projected/539635d3-17c9-43df-8080-8cb6b139e44a-kube-api-access-rzn4v\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.064187 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.064109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.064231 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.064184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/539635d3-17c9-43df-8080-8cb6b139e44a-openshift-service-ca\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.064267 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.064245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-maas-api-tls\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.165013 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.164985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-maas-api-tls\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.165141 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.165056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzn4v\" (UniqueName: \"kubernetes.io/projected/539635d3-17c9-43df-8080-8cb6b139e44a-kube-api-access-rzn4v\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.165141 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.165090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.165141 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.165123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/539635d3-17c9-43df-8080-8cb6b139e44a-openshift-service-ca\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.165287 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:32.165200 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 09 23:26:32.165287 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:32.165284 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs podName:539635d3-17c9-43df-8080-8cb6b139e44a nodeName:}" failed. No retries permitted until 2026-09-09 23:26:32.665262661 +0000 UTC m=+586.477871015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs") pod "maas-api-596f8fb547-x9zc5" (UID: "539635d3-17c9-43df-8080-8cb6b139e44a") : secret "maas-api-metrics-tls" not found Sep 09 23:26:32.165981 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.165955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/539635d3-17c9-43df-8080-8cb6b139e44a-openshift-service-ca\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.167605 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.167573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-maas-api-tls\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.175185 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.175165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzn4v\" (UniqueName: \"kubernetes.io/projected/539635d3-17c9-43df-8080-8cb6b139e44a-kube-api-access-rzn4v\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.670506 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:32.670466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:32.670695 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:32.670620 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 09 23:26:32.670695 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:32.670694 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs podName:539635d3-17c9-43df-8080-8cb6b139e44a nodeName:}" failed. No retries permitted until 2026-09-09 23:26:33.670674211 +0000 UTC m=+587.483282566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs") pod "maas-api-596f8fb547-x9zc5" (UID: "539635d3-17c9-43df-8080-8cb6b139e44a") : secret "maas-api-metrics-tls" not found Sep 09 23:26:33.677937 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:33.677889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:33.680248 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:33.680224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/539635d3-17c9-43df-8080-8cb6b139e44a-metrics-certs\") pod \"maas-api-596f8fb547-x9zc5\" (UID: \"539635d3-17c9-43df-8080-8cb6b139e44a\") " pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:33.780774 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:33.780747 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:34.051129 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:34.051102 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5"] Sep 09 23:26:34.052822 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:26:34.052790 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539635d3_17c9_43df_8080_8cb6b139e44a.slice/crio-fb73f7296a56a8620764b9043d1f800cd633017527ccb0823a7bb3aac60c6749 WatchSource:0}: Error finding container fb73f7296a56a8620764b9043d1f800cd633017527ccb0823a7bb3aac60c6749: Status 404 returned error can't find the container with id fb73f7296a56a8620764b9043d1f800cd633017527ccb0823a7bb3aac60c6749 Sep 09 23:26:34.419201 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:34.419164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" event={"ID":"539635d3-17c9-43df-8080-8cb6b139e44a","Type":"ContainerStarted","Data":"fb73f7296a56a8620764b9043d1f800cd633017527ccb0823a7bb3aac60c6749"} Sep 09 23:26:36.431927 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:36.431889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" event={"ID":"539635d3-17c9-43df-8080-8cb6b139e44a","Type":"ContainerStarted","Data":"a287d924877d37863b87425b9d544d746fc3c150810b650938c87c9d4c26bd96"} Sep 09 23:26:36.432300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:36.431975 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:36.450180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:36.450132 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" podStartSLOduration=3.835323197 podStartE2EDuration="5.450118201s" totalImagesPullingTime="1.614795004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:26:36.447637844 +0000 UTC m=+590.260246207" watchObservedRunningTime="2026-09-09 23:26:36.450118201 +0000 UTC m=+590.262726563" Sep 09 23:26:42.442168 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:42.442134 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-596f8fb547-x9zc5" Sep 09 23:26:46.584266 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:46.584233 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:26:46.585243 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:46.585222 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:26:46.598954 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:46.593838 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:26:46.599926 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:46.599899 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:26:49.599015 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.598976 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z"] Sep 09 23:26:49.604258 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.604237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.607288 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.607101 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pq5fv\"" Sep 09 23:26:49.607288 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.607179 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 23:26:49.607481 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.607413 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 09 23:26:49.607663 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.607646 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 23:26:49.612006 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.611975 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z"] Sep 09 23:26:49.711084 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.711049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.711267 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.711109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.711267 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.711170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.711267 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.711206 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c859fe39-ec97-4c58-b66d-4cf779dd3e21-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.711267 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.711232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86mp\" (UniqueName: \"kubernetes.io/projected/c859fe39-ec97-4c58-b66d-4cf779dd3e21-kube-api-access-t86mp\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.711456 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.711339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.811913 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.811878 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812090 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.811932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812090 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.811996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812090 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.812020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812090 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.812047 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c859fe39-ec97-4c58-b66d-4cf779dd3e21-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812334 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.812113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86mp\" (UniqueName: \"kubernetes.io/projected/c859fe39-ec97-4c58-b66d-4cf779dd3e21-kube-api-access-t86mp\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812396 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.812358 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-model-cache\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812459 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.812400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-home\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.812547 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.812526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.814339 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.814308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c859fe39-ec97-4c58-b66d-4cf779dd3e21-dshm\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.814496 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.814479 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c859fe39-ec97-4c58-b66d-4cf779dd3e21-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.819903 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.819882 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86mp\" (UniqueName: \"kubernetes.io/projected/c859fe39-ec97-4c58-b66d-4cf779dd3e21-kube-api-access-t86mp\") pod \"e2e-distinct-2-simulated-kserve-859856b874-g8k9z\" (UID: \"c859fe39-ec97-4c58-b66d-4cf779dd3e21\") " pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:49.918226 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:49.918191 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:50.199672 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:50.199517 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z"] Sep 09 23:26:50.202478 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:26:50.202444 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc859fe39_ec97_4c58_b66d_4cf779dd3e21.slice/crio-685e8f2c4ed053c2bf17cf1ceeee9bd5e999566c129156242c8525f6eceb016b WatchSource:0}: Error finding container 685e8f2c4ed053c2bf17cf1ceeee9bd5e999566c129156242c8525f6eceb016b: Status 404 returned error can't find the container with id 685e8f2c4ed053c2bf17cf1ceeee9bd5e999566c129156242c8525f6eceb016b Sep 09 23:26:50.499699 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:50.499609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" event={"ID":"c859fe39-ec97-4c58-b66d-4cf779dd3e21","Type":"ContainerStarted","Data":"685e8f2c4ed053c2bf17cf1ceeee9bd5e999566c129156242c8525f6eceb016b"} Sep 09 23:26:51.506244 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:51.506217 2559 generic.go:356] "Generic (PLEG): container finished" podID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" containerID="858c2c5f83a502956fe82eca7947a2dd06d333e274ee055043d5e6d255dcc75d" exitCode=0 Sep 09 23:26:51.506546 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:51.506299 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" event={"ID":"c859fe39-ec97-4c58-b66d-4cf779dd3e21","Type":"ContainerDied","Data":"858c2c5f83a502956fe82eca7947a2dd06d333e274ee055043d5e6d255dcc75d"} Sep 09 23:26:51.506546 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:51.506516 2559 scope.go:117] "RemoveContainer" containerID="858c2c5f83a502956fe82eca7947a2dd06d333e274ee055043d5e6d255dcc75d" Sep 09 23:26:52.512149 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:52.512118 2559 generic.go:356] "Generic (PLEG): container finished" podID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" containerID="75450306b5a93e271d6c5b6ab000feb2feefa7bbe49e37ce9f3988f8701c62ca" exitCode=0 Sep 09 23:26:52.512628 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:52.512163 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" event={"ID":"c859fe39-ec97-4c58-b66d-4cf779dd3e21","Type":"ContainerDied","Data":"75450306b5a93e271d6c5b6ab000feb2feefa7bbe49e37ce9f3988f8701c62ca"} Sep 09 23:26:52.512628 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:52.512239 2559 scope.go:117] "RemoveContainer" containerID="858c2c5f83a502956fe82eca7947a2dd06d333e274ee055043d5e6d255dcc75d" Sep 09 23:26:52.512628 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:52.512503 2559 scope.go:117] "RemoveContainer" containerID="75450306b5a93e271d6c5b6ab000feb2feefa7bbe49e37ce9f3988f8701c62ca" Sep 09 23:26:52.512764 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:52.512698 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:26:53.518768 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:53.518744 2559 scope.go:117] "RemoveContainer" containerID="75450306b5a93e271d6c5b6ab000feb2feefa7bbe49e37ce9f3988f8701c62ca" Sep 09 23:26:53.519105 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:53.518902 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:26:59.919039 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:59.919001 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:59.919506 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:59.919073 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:59.919506 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:59.919095 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:59.919506 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:59.919114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:26:59.919506 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:26:59.919475 2559 scope.go:117] "RemoveContainer" containerID="75450306b5a93e271d6c5b6ab000feb2feefa7bbe49e37ce9f3988f8701c62ca" Sep 09 23:26:59.919695 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:26:59.919675 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:27:12.201323 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.201250 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt"] Sep 09 23:27:12.203735 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.203715 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.206289 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.206270 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 09 23:27:12.215468 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.215445 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt"] Sep 09 23:27:12.304372 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.304340 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f69d50-0c61-4f0e-8331-05447c6be0f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.304564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.304391 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.304564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.304426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.304564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.304449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5t8l\" (UniqueName: \"kubernetes.io/projected/f9f69d50-0c61-4f0e-8331-05447c6be0f1-kube-api-access-m5t8l\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.304564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.304480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.304564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.304508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.405644 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.405610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.405644 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.405648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.405827 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.405665 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5t8l\" (UniqueName: \"kubernetes.io/projected/f9f69d50-0c61-4f0e-8331-05447c6be0f1-kube-api-access-m5t8l\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.405827 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.405688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.405827 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.405711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.405827 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.405812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f69d50-0c61-4f0e-8331-05447c6be0f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.406115 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.406092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.406207 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.406130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.406207 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.406170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.407870 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.407850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f69d50-0c61-4f0e-8331-05447c6be0f1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.408149 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.408132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f69d50-0c61-4f0e-8331-05447c6be0f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.413189 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.413161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5t8l\" (UniqueName: \"kubernetes.io/projected/f9f69d50-0c61-4f0e-8331-05447c6be0f1-kube-api-access-m5t8l\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt\" (UID: \"f9f69d50-0c61-4f0e-8331-05447c6be0f1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.515060 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.514999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:12.786448 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:12.786424 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt"] Sep 09 23:27:12.788009 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:27:12.787981 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f69d50_0c61_4f0e_8331_05447c6be0f1.slice/crio-b07a0518053d36f2af5924bd20c4313480f58d0b7e4c8c6e934acc972596e057 WatchSource:0}: Error finding container b07a0518053d36f2af5924bd20c4313480f58d0b7e4c8c6e934acc972596e057: Status 404 returned error can't find the container with id b07a0518053d36f2af5924bd20c4313480f58d0b7e4c8c6e934acc972596e057 Sep 09 23:27:13.611161 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:13.611127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" event={"ID":"f9f69d50-0c61-4f0e-8331-05447c6be0f1","Type":"ContainerStarted","Data":"132e02237f1b5b7d58b61c7363b0a91e01490c14bb1ac569ce394b9b7bec1ed2"} Sep 09 23:27:13.611524 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:13.611177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" event={"ID":"f9f69d50-0c61-4f0e-8331-05447c6be0f1","Type":"ContainerStarted","Data":"b07a0518053d36f2af5924bd20c4313480f58d0b7e4c8c6e934acc972596e057"} Sep 09 23:27:13.631939 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:13.631888 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" podStartSLOduration=1.4703130309999999 podStartE2EDuration="1.631874701s" totalImagesPullingTime="161.56167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:27:13.630818526 +0000 UTC m=+627.443426889" watchObservedRunningTime="2026-09-09 23:27:13.631874701 +0000 UTC m=+627.444483063" Sep 09 23:27:14.638319 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:14.638287 2559 scope.go:117] "RemoveContainer" containerID="75450306b5a93e271d6c5b6ab000feb2feefa7bbe49e37ce9f3988f8701c62ca" Sep 09 23:27:15.622574 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:15.622538 2559 generic.go:356] "Generic (PLEG): container finished" podID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" containerID="517384e8d524fa6c0676bd1abe47088708dc5acbe36218533277af203d178529" exitCode=0 Sep 09 23:27:15.622764 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:15.622614 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" event={"ID":"c859fe39-ec97-4c58-b66d-4cf779dd3e21","Type":"ContainerDied","Data":"517384e8d524fa6c0676bd1abe47088708dc5acbe36218533277af203d178529"} Sep 09 23:27:15.622764 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:15.622672 2559 scope.go:117] "RemoveContainer" containerID="75450306b5a93e271d6c5b6ab000feb2feefa7bbe49e37ce9f3988f8701c62ca" Sep 09 23:27:15.623052 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:15.623035 2559 scope.go:117] "RemoveContainer" containerID="517384e8d524fa6c0676bd1abe47088708dc5acbe36218533277af203d178529" Sep 09 23:27:15.623263 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:15.623242 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:27:19.919019 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:19.918982 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:19.919448 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:19.919042 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:19.919448 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:19.919059 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:19.919448 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:19.919073 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:19.919448 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:19.919407 2559 scope.go:117] "RemoveContainer" containerID="517384e8d524fa6c0676bd1abe47088708dc5acbe36218533277af203d178529" Sep 09 23:27:19.919653 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:19.919623 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:27:22.515369 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:22.515319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:22.515797 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:22.515399 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:22.527848 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:22.527825 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:22.674180 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:22.674151 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt" Sep 09 23:27:33.638009 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:33.637977 2559 scope.go:117] "RemoveContainer" containerID="517384e8d524fa6c0676bd1abe47088708dc5acbe36218533277af203d178529" Sep 09 23:27:33.643057 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:33.638169 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:27:36.109946 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.109911 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr"] Sep 09 23:27:36.113928 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.113902 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.116625 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.116583 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 09 23:27:36.124477 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.124453 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr"] Sep 09 23:27:36.198864 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.198836 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfbjm\" (UniqueName: \"kubernetes.io/projected/add1ed20-5ff1-4ac4-ad52-773ec32d93db-kube-api-access-nfbjm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.199000 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.198879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.199000 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.198906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add1ed20-5ff1-4ac4-ad52-773ec32d93db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.199000 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.198978 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.199169 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.199055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.199169 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.199088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300208 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300339 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300339 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300245 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfbjm\" (UniqueName: \"kubernetes.io/projected/add1ed20-5ff1-4ac4-ad52-773ec32d93db-kube-api-access-nfbjm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300339 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300339 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add1ed20-5ff1-4ac4-ad52-773ec32d93db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300339 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300313 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300681 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300656 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300766 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.300766 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.300697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.302577 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.302557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1ed20-5ff1-4ac4-ad52-773ec32d93db-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.302872 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.302852 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add1ed20-5ff1-4ac4-ad52-773ec32d93db-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.311603 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.311574 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfbjm\" (UniqueName: \"kubernetes.io/projected/add1ed20-5ff1-4ac4-ad52-773ec32d93db-kube-api-access-nfbjm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr\" (UID: \"add1ed20-5ff1-4ac4-ad52-773ec32d93db\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.426016 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.425995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:36.700296 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.700268 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr"] Sep 09 23:27:36.701826 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:27:36.701802 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd1ed20_5ff1_4ac4_ad52_773ec32d93db.slice/crio-aca4060ea3b6bb6afe631f390cd8204cb1e48720af8e4431b0354f5bb69acdde WatchSource:0}: Error finding container aca4060ea3b6bb6afe631f390cd8204cb1e48720af8e4431b0354f5bb69acdde: Status 404 returned error can't find the container with id aca4060ea3b6bb6afe631f390cd8204cb1e48720af8e4431b0354f5bb69acdde Sep 09 23:27:36.726642 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:36.726615 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" event={"ID":"add1ed20-5ff1-4ac4-ad52-773ec32d93db","Type":"ContainerStarted","Data":"aca4060ea3b6bb6afe631f390cd8204cb1e48720af8e4431b0354f5bb69acdde"} Sep 09 23:27:37.733170 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:37.733138 2559 generic.go:356] "Generic (PLEG): container finished" podID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" containerID="3fe47d421396e03c9a0b775c4e118b6ed411651784e2a2114b312b35e6f0b827" exitCode=0 Sep 09 23:27:37.733552 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:37.733225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" event={"ID":"add1ed20-5ff1-4ac4-ad52-773ec32d93db","Type":"ContainerDied","Data":"3fe47d421396e03c9a0b775c4e118b6ed411651784e2a2114b312b35e6f0b827"} Sep 09 23:27:37.733552 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:37.733432 2559 scope.go:117] "RemoveContainer" containerID="3fe47d421396e03c9a0b775c4e118b6ed411651784e2a2114b312b35e6f0b827" Sep 09 23:27:38.739413 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:38.739377 2559 generic.go:356] "Generic (PLEG): container finished" podID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" containerID="aa5011f6a42185173389aa8474c0ae293eb62caa37ddc38a5fc09b6824d6d710" exitCode=0 Sep 09 23:27:38.739883 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:38.739432 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" event={"ID":"add1ed20-5ff1-4ac4-ad52-773ec32d93db","Type":"ContainerDied","Data":"aa5011f6a42185173389aa8474c0ae293eb62caa37ddc38a5fc09b6824d6d710"} Sep 09 23:27:38.739883 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:38.739492 2559 scope.go:117] "RemoveContainer" containerID="3fe47d421396e03c9a0b775c4e118b6ed411651784e2a2114b312b35e6f0b827" Sep 09 23:27:38.739883 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:38.739818 2559 scope.go:117] "RemoveContainer" containerID="aa5011f6a42185173389aa8474c0ae293eb62caa37ddc38a5fc09b6824d6d710" Sep 09 23:27:38.740045 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:38.740026 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:27:39.746077 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:39.746045 2559 scope.go:117] "RemoveContainer" containerID="aa5011f6a42185173389aa8474c0ae293eb62caa37ddc38a5fc09b6824d6d710" Sep 09 23:27:39.746522 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:39.746257 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:27:45.638789 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:45.638758 2559 scope.go:117] "RemoveContainer" containerID="517384e8d524fa6c0676bd1abe47088708dc5acbe36218533277af203d178529" Sep 09 23:27:45.639167 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:45.639035 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:27:46.426306 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.426269 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:46.426467 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.426329 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:46.426467 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.426346 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:46.426467 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.426359 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:27:46.426722 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.426709 2559 scope.go:117] "RemoveContainer" containerID="aa5011f6a42185173389aa8474c0ae293eb62caa37ddc38a5fc09b6824d6d710" Sep 09 23:27:46.426887 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:46.426866 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:27:46.780336 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.780256 2559 generic.go:356] "Generic (PLEG): container finished" podID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" containerID="292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1" exitCode=0 Sep 09 23:27:46.780336 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.780296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" event={"ID":"c859fe39-ec97-4c58-b66d-4cf779dd3e21","Type":"ContainerDied","Data":"292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1"} Sep 09 23:27:46.780829 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.780368 2559 scope.go:117] "RemoveContainer" containerID="517384e8d524fa6c0676bd1abe47088708dc5acbe36218533277af203d178529" Sep 09 23:27:46.780829 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:46.780741 2559 scope.go:117] "RemoveContainer" containerID="292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1" Sep 09 23:27:46.780965 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:46.780943 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:27:49.918282 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:49.918245 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:49.918658 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:49.918309 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:49.918658 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:49.918327 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:49.918658 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:49.918341 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:27:49.918763 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:27:49.918687 2559 scope.go:117] "RemoveContainer" containerID="292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1" Sep 09 23:27:49.918906 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:27:49.918882 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:28:01.638045 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:01.638013 2559 scope.go:117] "RemoveContainer" containerID="aa5011f6a42185173389aa8474c0ae293eb62caa37ddc38a5fc09b6824d6d710" Sep 09 23:28:02.638436 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:02.638405 2559 scope.go:117] "RemoveContainer" containerID="292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1" Sep 09 23:28:02.638924 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:02.638556 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:28:02.858815 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:02.858782 2559 generic.go:356] "Generic (PLEG): container finished" podID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" containerID="fb60c04338ddfeb4a8d521b712c0150cb439f65420075067186cae86dc66e38c" exitCode=0 Sep 09 23:28:02.858988 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:02.858855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" event={"ID":"add1ed20-5ff1-4ac4-ad52-773ec32d93db","Type":"ContainerDied","Data":"fb60c04338ddfeb4a8d521b712c0150cb439f65420075067186cae86dc66e38c"} Sep 09 23:28:02.858988 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:02.858915 2559 scope.go:117] "RemoveContainer" containerID="aa5011f6a42185173389aa8474c0ae293eb62caa37ddc38a5fc09b6824d6d710" Sep 09 23:28:02.859360 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:02.859343 2559 scope.go:117] "RemoveContainer" containerID="fb60c04338ddfeb4a8d521b712c0150cb439f65420075067186cae86dc66e38c" Sep 09 23:28:02.859549 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:02.859530 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:28:06.426430 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:06.426383 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:06.426859 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:06.426457 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:06.426859 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:06.426479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:06.426859 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:06.426501 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:06.426963 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:06.426862 2559 scope.go:117] "RemoveContainer" containerID="fb60c04338ddfeb4a8d521b712c0150cb439f65420075067186cae86dc66e38c" Sep 09 23:28:06.427056 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:06.427036 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:28:17.638183 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:17.638150 2559 scope.go:117] "RemoveContainer" containerID="292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1" Sep 09 23:28:17.638682 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:17.638370 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:28:20.639028 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:20.638997 2559 scope.go:117] "RemoveContainer" containerID="fb60c04338ddfeb4a8d521b712c0150cb439f65420075067186cae86dc66e38c" Sep 09 23:28:20.639391 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:20.639166 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:28:29.638839 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:29.638808 2559 scope.go:117] "RemoveContainer" containerID="292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1" Sep 09 23:28:29.985277 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:29.985246 2559 generic.go:356] "Generic (PLEG): container finished" podID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" exitCode=0 Sep 09 23:28:29.985431 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:29.985320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" event={"ID":"c859fe39-ec97-4c58-b66d-4cf779dd3e21","Type":"ContainerDied","Data":"a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5"} Sep 09 23:28:29.985431 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:29.985392 2559 scope.go:117] "RemoveContainer" containerID="292ed9735ee218a10119aa55cb3f1d39fd8e0e437f109f41716553154fb741f1" Sep 09 23:28:29.985791 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:29.985768 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:28:29.985977 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:29.985954 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:28:32.644817 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:32.644788 2559 scope.go:117] "RemoveContainer" containerID="fb60c04338ddfeb4a8d521b712c0150cb439f65420075067186cae86dc66e38c" Sep 09 23:28:33.001952 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:33.001922 2559 generic.go:356] "Generic (PLEG): container finished" podID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" containerID="c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d" exitCode=0 Sep 09 23:28:33.002069 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:33.001990 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" event={"ID":"add1ed20-5ff1-4ac4-ad52-773ec32d93db","Type":"ContainerDied","Data":"c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d"} Sep 09 23:28:33.002069 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:33.002059 2559 scope.go:117] "RemoveContainer" containerID="fb60c04338ddfeb4a8d521b712c0150cb439f65420075067186cae86dc66e38c" Sep 09 23:28:33.002369 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:33.002354 2559 scope.go:117] "RemoveContainer" containerID="c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d" Sep 09 23:28:33.002568 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:33.002548 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:28:36.426493 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:36.426436 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:36.426877 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:36.426549 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:36.426877 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:36.426568 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:36.426877 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:36.426580 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:28:36.427005 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:36.426905 2559 scope.go:117] "RemoveContainer" containerID="c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d" Sep 09 23:28:36.427074 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:36.427057 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:28:39.918333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:39.918258 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:28:39.918333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:39.918303 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:28:39.918333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:39.918319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:28:39.918333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:39.918332 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:28:39.918782 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:39.918634 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:28:39.918819 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:39.918793 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:28:48.638274 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:48.638239 2559 scope.go:117] "RemoveContainer" containerID="c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d" Sep 09 23:28:48.638791 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:48.638425 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:28:52.638373 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:28:52.638335 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:28:52.638820 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:28:52.638618 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:29:03.638248 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:03.638209 2559 scope.go:117] "RemoveContainer" containerID="c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d" Sep 09 23:29:03.638741 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:03.638447 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:29:06.640300 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:06.640273 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:29:06.640673 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:06.640428 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:29:17.638503 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:17.638478 2559 scope.go:117] "RemoveContainer" containerID="c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d" Sep 09 23:29:18.213425 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:18.213393 2559 generic.go:356] "Generic (PLEG): container finished" podID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" exitCode=0 Sep 09 23:29:18.213565 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:18.213464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" event={"ID":"add1ed20-5ff1-4ac4-ad52-773ec32d93db","Type":"ContainerDied","Data":"34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57"} Sep 09 23:29:18.213565 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:18.213545 2559 scope.go:117] "RemoveContainer" containerID="c3e30e3be717bd5ba50cdb04f12841bea75b903d9c03f6a8763d8f004161267d" Sep 09 23:29:18.213957 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:18.213939 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:29:18.214122 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:18.214097 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:29:19.637949 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:19.637918 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:29:19.638383 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:19.638076 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:29:26.426829 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:26.426791 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:29:26.427213 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:26.426853 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:29:26.427213 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:26.426870 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:29:26.427213 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:26.426883 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:29:26.427213 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:26.427196 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:29:26.427384 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:26.427365 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:29:34.638942 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:34.638913 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:29:34.639391 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:34.639138 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:29:39.637946 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:39.637914 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:29:39.638327 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:39.638087 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:29:45.638614 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:45.638559 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:29:45.638994 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:45.638739 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:29:52.638872 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:52.638840 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:29:52.639325 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:52.639016 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:29:58.638912 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:58.638884 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:29:59.415267 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.415237 2559 generic.go:356] "Generic (PLEG): container finished" podID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" exitCode=0 Sep 09 23:29:59.415428 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.415306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" event={"ID":"c859fe39-ec97-4c58-b66d-4cf779dd3e21","Type":"ContainerDied","Data":"676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754"} Sep 09 23:29:59.415428 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.415362 2559 scope.go:117] "RemoveContainer" containerID="a57cdc426eeda259b96ac71d405922d5ea598ef831095939fccac705e7f2afa5" Sep 09 23:29:59.415649 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.415634 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:29:59.415894 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:29:59.415866 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:29:59.918483 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.918448 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:29:59.918887 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.918510 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:29:59.918887 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.918551 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:29:59.918887 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:29:59.918574 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" Sep 09 23:30:00.422012 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:00.421977 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:30:00.422187 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:00.422169 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:30:01.426606 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:01.426558 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:30:01.426970 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:01.426769 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:30:06.641652 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:06.641621 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:30:06.642059 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:06.641815 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:30:15.638515 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:15.638445 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:30:15.638885 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:15.638640 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:30:17.638758 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:17.638729 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:30:17.639202 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:17.638955 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:30:28.638423 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:28.638384 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:30:28.638919 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:28.638630 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:30:30.643882 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:30.643853 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:30:30.644251 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:30.644016 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:30:41.638665 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:41.638635 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:30:42.619221 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:42.619186 2559 generic.go:356] "Generic (PLEG): container finished" podID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" exitCode=0 Sep 09 23:30:42.619408 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:42.619237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" event={"ID":"add1ed20-5ff1-4ac4-ad52-773ec32d93db","Type":"ContainerDied","Data":"57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d"} Sep 09 23:30:42.619408 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:42.619305 2559 scope.go:117] "RemoveContainer" containerID="34f9bb29891664bb5321044f7e6619839192b121f1b2fcad2bacd5c92305aa57" Sep 09 23:30:42.619702 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:42.619681 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:30:42.619895 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:42.619872 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:30:45.639031 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:45.639004 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:30:45.639393 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:45.639177 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:30:46.426499 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:46.426466 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:30:46.426692 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:46.426521 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:30:46.426692 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:46.426538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:30:46.426692 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:46.426551 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" Sep 09 23:30:46.426880 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:46.426867 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:30:46.427075 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:46.427057 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:30:58.638564 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:30:58.638531 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:30:58.639037 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:30:58.638744 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:31:00.637989 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:00.637958 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:31:00.638356 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:00.638145 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:31:12.638999 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:12.638968 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:31:12.639371 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:12.639146 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:31:13.638205 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:13.638173 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:31:13.638434 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:13.638405 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:31:24.638896 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:24.638817 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:31:24.639251 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:24.638978 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:31:26.640604 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:26.640566 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:31:26.640989 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:26.640758 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:31:38.638336 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:38.638303 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:31:38.638950 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:38.638456 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:31:38.638950 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:38.638480 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:31:38.638950 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:38.638639 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:31:46.551841 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:46.551769 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_add1ed20-5ff1-4ac4-ad52-773ec32d93db/main/5.log" Sep 09 23:31:46.622459 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:46.622436 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:31:46.623663 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:46.623640 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:31:46.632963 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:46.632947 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:31:46.634406 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:46.634386 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:31:46.659766 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:46.659744 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_add1ed20-5ff1-4ac4-ad52-773ec32d93db/main/5.log" Sep 09 23:31:50.638757 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:50.638722 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:31:50.639124 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:50.638905 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:31:52.844231 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:52.844202 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qxxjp_8369569d-4f02-4ef0-8cb8-fcc9e129b5a2/manager/0.log" Sep 09 23:31:52.980891 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:52.980859 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9bdb9bd9b-qrcx8_077c5ec5-8276-4918-9168-8ffd8cf9531c/manager/0.log" Sep 09 23:31:53.302078 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:53.302050 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c5c8967d-hptqg_b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f/copy-manifests/0.log" Sep 09 23:31:53.320251 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:53.320218 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c5c8967d-hptqg_b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f/manager/0.log" Sep 09 23:31:53.638816 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:53.638740 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:31:53.638955 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:31:53.638909 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:31:53.861305 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:53.861278 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-596f8fb547-x9zc5_539635d3-17c9-43df-8080-8cb6b139e44a/maas-api/0.log" Sep 09 23:31:54.073400 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:54.073361 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lt9mq_cbaeb6d0-6212-4d14-a561-3990ec5befdd/postgres/0.log" Sep 09 23:31:55.250163 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:55.250131 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rp4t4_94501049-55f0-4319-9716-b1cb106304f2/manager/0.log" Sep 09 23:31:56.550769 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:56.550739 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f765574b6-292xd_da17354a-6849-474a-95c0-10c651c10d53/router/0.log" Sep 09 23:31:56.878502 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:56.878424 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-859856b874-g8k9z_c859fe39-ec97-4c58-b66d-4cf779dd3e21/main/5.log" Sep 09 23:31:57.088406 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:57.088376 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-hwbrt_f9f69d50-0c61-4f0e-8331-05447c6be0f1/main/0.log" Sep 09 23:31:57.410763 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:31:57.410732 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_add1ed20-5ff1-4ac4-ad52-773ec32d93db/main/5.log" Sep 09 23:32:01.638396 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:01.638366 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:32:01.638769 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:32:01.638545 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:32:05.805932 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:05.805898 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5rgz4_1a33b8ed-f089-4b62-b83c-23c8db52500b/global-pull-secret-syncer/0.log" Sep 09 23:32:05.973311 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:05.973281 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lb6gg_27ef369c-d564-486b-af79-a543ba16dc83/konnectivity-agent/0.log" Sep 09 23:32:06.031324 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:06.031300 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-167.ec2.internal_55584c1d8f97229f4aae3d06860d9639/haproxy/0.log" Sep 09 23:32:07.638270 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:07.638232 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:32:07.638717 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:32:07.638453 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:32:10.705837 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:10.705800 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rp4t4_94501049-55f0-4319-9716-b1cb106304f2/manager/0.log" Sep 09 23:32:12.375523 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:12.375497 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-kh9wq_7cc9c356-f4bd-409d-801a-d19df7559c6c/cluster-monitoring-operator/0.log" Sep 09 23:32:12.619475 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:12.619447 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxzrg_e5e12116-c427-4942-893d-ef88d9af2f06/node-exporter/0.log" Sep 09 23:32:12.636877 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:12.636808 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxzrg_e5e12116-c427-4942-893d-ef88d9af2f06/kube-rbac-proxy/0.log" Sep 09 23:32:12.638429 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:12.638410 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:32:12.638629 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:32:12.638583 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:32:12.653312 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:12.653291 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxzrg_e5e12116-c427-4942-893d-ef88d9af2f06/init-textfile/0.log" Sep 09 23:32:13.093130 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:13.093095 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-dfwzp_8bf26d86-df94-4685-8519-92a5fa9dd1f3/prometheus-operator-admission-webhook/0.log" Sep 09 23:32:14.679985 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:14.679953 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-v5mr2_51a700c9-334f-4769-84f3-5e806b045aad/networking-console-plugin/0.log" Sep 09 23:32:15.278615 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:15.278567 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/1.log" Sep 09 23:32:15.287366 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:15.287340 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-wq44d_9a26c0c0-6ec6-4b20-8bb7-b1dcc9fa27e5/console-operator/2.log" Sep 09 23:32:15.905381 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:15.905344 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-xb6wn_b1927131-fc1c-4fe7-a392-8400e1d6b6b7/download-server/0.log" Sep 09 23:32:16.256578 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.256541 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6"] Sep 09 23:32:16.260045 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.260022 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.262862 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.262840 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pwd2c\"/\"kube-root-ca.crt\"" Sep 09 23:32:16.262959 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.262871 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pwd2c\"/\"openshift-service-ca.crt\"" Sep 09 23:32:16.264111 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.264090 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pwd2c\"/\"default-dockercfg-nkb97\"" Sep 09 23:32:16.269919 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.269901 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6"] Sep 09 23:32:16.306135 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.306107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-podres\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.306308 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.306154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-lib-modules\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.306308 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.306199 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6b8\" (UniqueName: \"kubernetes.io/projected/30d71fae-ca5b-4632-9865-3f4ebc4d682b-kube-api-access-dz6b8\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.306308 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.306239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-sys\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.306457 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.306342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-proc\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407280 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-proc\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407496 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-podres\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407496 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-lib-modules\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407496 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6b8\" (UniqueName: \"kubernetes.io/projected/30d71fae-ca5b-4632-9865-3f4ebc4d682b-kube-api-access-dz6b8\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407496 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-proc\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407496 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407384 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-sys\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407496 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-sys\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407762 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-lib-modules\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.407762 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.407512 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30d71fae-ca5b-4632-9865-3f4ebc4d682b-podres\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.415170 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.415154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6b8\" (UniqueName: \"kubernetes.io/projected/30d71fae-ca5b-4632-9865-3f4ebc4d682b-kube-api-access-dz6b8\") pod \"perf-node-gather-daemonset-jxdh6\" (UID: \"30d71fae-ca5b-4632-9865-3f4ebc4d682b\") " pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.570977 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.570878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:16.634172 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.634111 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-9sz65_4711a9be-591a-4346-a3ca-15156411555d/volume-data-source-validator/0.log" Sep 09 23:32:16.839065 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:16.839038 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6"] Sep 09 23:32:16.839672 ip-10-0-142-167 kubenswrapper[2559]: W0909 23:32:16.839648 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod30d71fae_ca5b_4632_9865_3f4ebc4d682b.slice/crio-84f06b4b171fb36e6ebc0db7d47be60c84069adce71e91219635dd7ef06a2596 WatchSource:0}: Error finding container 84f06b4b171fb36e6ebc0db7d47be60c84069adce71e91219635dd7ef06a2596: Status 404 returned error can't find the container with id 84f06b4b171fb36e6ebc0db7d47be60c84069adce71e91219635dd7ef06a2596 Sep 09 23:32:17.060007 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:17.059966 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" event={"ID":"30d71fae-ca5b-4632-9865-3f4ebc4d682b","Type":"ContainerStarted","Data":"3299363d09ddf998fe11af3986d70d7c110424897267ebf9b3997577ab59359e"} Sep 09 23:32:17.060418 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:17.060028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" event={"ID":"30d71fae-ca5b-4632-9865-3f4ebc4d682b","Type":"ContainerStarted","Data":"84f06b4b171fb36e6ebc0db7d47be60c84069adce71e91219635dd7ef06a2596"} Sep 09 23:32:17.060418 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:17.060056 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:17.076176 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:17.076137 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" podStartSLOduration=1.076123167 podStartE2EDuration="1.076123167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:32:17.075756114 +0000 UTC m=+930.888364474" watchObservedRunningTime="2026-09-09 23:32:17.076123167 +0000 UTC m=+930.888731528" Sep 09 23:32:17.874426 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:17.874390 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8jpxs_6c5f9dc9-9bed-402b-af9f-16b918a054e5/dns/0.log" Sep 09 23:32:17.892790 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:17.892768 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8jpxs_6c5f9dc9-9bed-402b-af9f-16b918a054e5/kube-rbac-proxy/0.log" Sep 09 23:32:17.912297 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:17.912272 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b2z6r_633f88c9-63c9-48d4-9ffc-523090d75649/dns-node-resolver/0.log" Sep 09 23:32:18.495619 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:18.495574 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ld9zs_70c765f6-1f3d-4660-bc36-48293230f6e3/node-ca/0.log" Sep 09 23:32:19.590839 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:19.590803 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f765574b6-292xd_da17354a-6849-474a-95c0-10c651c10d53/router/0.log" Sep 09 23:32:20.179509 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:20.179473 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4x4rk_6698e90f-06ad-4a2e-af70-ee5821fa0a0d/serve-healthcheck-canary/0.log" Sep 09 23:32:20.638900 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:20.638819 2559 scope.go:117] "RemoveContainer" containerID="676aec977da028383425dab0a5c2d890561f3861ae8da7f48e1c62242e056754" Sep 09 23:32:20.639255 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:32:20.639009 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-2-simulated-kserve-859856b874-g8k9z_llm(c859fe39-ec97-4c58-b66d-4cf779dd3e21)\"" pod="llm/e2e-distinct-2-simulated-kserve-859856b874-g8k9z" podUID="c859fe39-ec97-4c58-b66d-4cf779dd3e21" Sep 09 23:32:20.670658 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:20.670623 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-sktw2_9eea67d0-b1b9-4305-811e-20aa6ea233ed/insights-operator/0.log" Sep 09 23:32:20.671062 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:20.671046 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-sktw2_9eea67d0-b1b9-4305-811e-20aa6ea233ed/insights-operator/1.log" Sep 09 23:32:20.758200 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:20.758174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pf8kf_42127793-921c-4220-b533-1797ac467275/kube-rbac-proxy/0.log" Sep 09 23:32:20.777221 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:20.777189 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pf8kf_42127793-921c-4220-b533-1797ac467275/exporter/0.log" Sep 09 23:32:20.796269 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:20.796242 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pf8kf_42127793-921c-4220-b533-1797ac467275/extractor/0.log" Sep 09 23:32:23.010860 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:23.010825 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qxxjp_8369569d-4f02-4ef0-8cb8-fcc9e129b5a2/manager/0.log" Sep 09 23:32:23.045267 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:23.045244 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9bdb9bd9b-qrcx8_077c5ec5-8276-4918-9168-8ffd8cf9531c/manager/0.log" Sep 09 23:32:23.073821 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:23.073801 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pwd2c/perf-node-gather-daemonset-jxdh6" Sep 09 23:32:23.183844 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:23.183817 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c5c8967d-hptqg_b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f/manager/0.log" Sep 09 23:32:23.200962 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:23.200944 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8c5c8967d-hptqg_b8a32b31-5731-4ea6-a9f4-e05eda6e6d5f/copy-manifests/0.log" Sep 09 23:32:24.405875 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:24.405841 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67998c845-lhwn7_451a1f57-5b9f-4902-bd9f-e5a16a20fb59/manager/0.log" Sep 09 23:32:25.638873 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:25.638843 2559 scope.go:117] "RemoveContainer" containerID="57d3b9da88f8800b73d46205f465f53e670e7945712751cbcd24879ba9b8b71d" Sep 09 23:32:25.639265 ip-10-0-142-167 kubenswrapper[2559]: E0909 23:32:25.639042 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr_llm(add1ed20-5ff1-4ac4-ad52-773ec32d93db)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-7jtcr" podUID="add1ed20-5ff1-4ac4-ad52-773ec32d93db" Sep 09 23:32:26.234914 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:26.234884 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f765574b6-292xd_da17354a-6849-474a-95c0-10c651c10d53/router/0.log" Sep 09 23:32:27.393362 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:27.393283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-f765574b6-292xd_da17354a-6849-474a-95c0-10c651c10d53/router/0.log" Sep 09 23:32:29.660380 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:29.660350 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kd55b_af5bcce7-094c-4f73-a233-d25bb0ecfb73/migrator/0.log" Sep 09 23:32:29.679853 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:29.679831 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-kd55b_af5bcce7-094c-4f73-a233-d25bb0ecfb73/graceful-termination/0.log" Sep 09 23:32:30.017281 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:30.017247 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-hrjpm_7abba9dc-6fbe-4927-8298-ddda486d8ed1/kube-storage-version-migrator-operator/1.log" Sep 09 23:32:30.018998 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:30.018968 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-hrjpm_7abba9dc-6fbe-4927-8298-ddda486d8ed1/kube-storage-version-migrator-operator/0.log" Sep 09 23:32:30.945858 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:30.945828 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-28zfb_14ee243b-7559-462e-9414-a47ed4df6877/kube-multus-additional-cni-plugins/0.log" Sep 09 23:32:30.966400 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:30.966371 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-28zfb_14ee243b-7559-462e-9414-a47ed4df6877/egress-router-binary-copy/0.log" Sep 09 23:32:30.985006 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:30.984979 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-28zfb_14ee243b-7559-462e-9414-a47ed4df6877/cni-plugins/0.log" Sep 09 23:32:31.003123 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:31.003103 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-28zfb_14ee243b-7559-462e-9414-a47ed4df6877/bond-cni-plugin/0.log" Sep 09 23:32:31.020746 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:31.020723 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-28zfb_14ee243b-7559-462e-9414-a47ed4df6877/routeoverride-cni/0.log" Sep 09 23:32:31.038582 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:31.038561 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-28zfb_14ee243b-7559-462e-9414-a47ed4df6877/whereabouts-cni-bincopy/0.log" Sep 09 23:32:31.057607 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:31.057577 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-28zfb_14ee243b-7559-462e-9414-a47ed4df6877/whereabouts-cni/0.log" Sep 09 23:32:31.429273 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:31.429243 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ss8v8_11c60a16-cbba-4a3b-b165-66da33d20d8b/kube-multus/0.log" Sep 09 23:32:31.447861 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:31.447836 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ht64b_060a36f0-058c-4110-a094-dff084254fb5/network-metrics-daemon/0.log" Sep 09 23:32:31.463945 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:31.463906 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ht64b_060a36f0-058c-4110-a094-dff084254fb5/kube-rbac-proxy/0.log" Sep 09 23:32:32.481333 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.481282 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-controller/0.log" Sep 09 23:32:32.497008 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.496976 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/1.log" Sep 09 23:32:32.497190 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.497071 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovn-acl-logging/0.log" Sep 09 23:32:32.514161 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.514139 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/kube-rbac-proxy-node/0.log" Sep 09 23:32:32.532124 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.532104 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 23:32:32.547694 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.547675 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/northd/0.log" Sep 09 23:32:32.564696 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.564677 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/nbdb/0.log" Sep 09 23:32:32.582277 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.582258 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/sbdb/0.log" Sep 09 23:32:32.635673 ip-10-0-142-167 kubenswrapper[2559]: I0909 23:32:32.635645 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b9zzg_22846b1d-d458-426a-9a93-0b2c598879c1/ovnkube-controller/0.log"