2026-07-06T22:01:53.197378088Z [disk usage checker] Started 2026-07-06T22:01:53.200695609Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-06T22:01:53.388852419Z Gathering data for ns/openshift-cluster-version... 2026-07-06T22:01:53.532618258Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-06T22:01:53.620877275Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T22:01:53.697539367Z Gathering data for ns/default... 2026-07-06T22:01:53.987028779Z Gathering data for ns/openshift... 2026-07-06T22:01:54.423092817Z Gathering data for ns/kube-system... 2026-07-06T22:01:55.112805880Z Gathering data for ns/openshift-etcd... 2026-07-06T22:01:55.437951569Z Wrote inspect data to must-gather. 2026-07-06T22:01:55.437980987Z error: inspection completed with the errors occurred while gathering data: 2026-07-06T22:01:55.437980987Z namespaces "assisted-installer" not found 2026-07-06T22:01:55.935198111Z Waiting on subprocesses to finish execution. 2026-07-06T22:01:55.943723540Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-06T22:01:55.950604588Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-06T22:01:55.958253780Z Executing Istio gather script 2026-07-06T22:01:55.959401580Z INFO: Gathering HAProxy config files 2026-07-06T22:01:55.996877195Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-06T22:01:56.015003417Z INFO: Collecting host service logs for crio 2026-07-06T22:01:56.015379858Z INFO: Collecting host service logs for kubelet 2026-07-06T22:01:56.015740864Z INFO: Collecting host service logs for rpm-ostreed 2026-07-06T22:01:56.016087099Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-06T22:01:56.016469564Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-06T22:01:56.016845096Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-06T22:01:56.017208029Z INFO: Collecting host service logs for NetworkManager 2026-07-06T22:01:56.017577236Z INFO: Collecting host service logs for openvswitch 2026-07-06T22:01:56.017928077Z INFO: Collecting host service logs for ovs-configuration 2026-07-06T22:01:56.018288549Z INFO: Collecting host service logs for ovsdb-server 2026-07-06T22:01:56.018665277Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-06T22:01:56.019091175Z INFO: Waiting for worker host service log collection to complete ... 2026-07-06T22:01:56.081687537Z INFO: Waiting for node performance related collection to complete ... 2026-07-06T22:01:56.346364933Z No resources found 2026-07-06T22:01:56.475876663Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-06T22:01:56.581084202Z No resources found in openshift-etcd namespace. 2026-07-06T22:01:56.704971768Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-cnsh7 2026-07-06T22:01:56.705021830Z insights-runtime-extractor-b9wqf 2026-07-06T22:01:56.705042300Z insights-runtime-extractor-hwvqm 2026-07-06T22:01:56.705062890Z insights-runtime-extractor-lsz7j 2026-07-06T22:01:57.102423237Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-06T22:01:57.102423237Z See 'oc rsync -h' for help and examples 2026-07-06T22:01:57.127815856Z INFO: "metallb-operator" not detected. Skipping. 2026-07-06T22:01:57.174501036Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-06T22:01:57.175112665Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-06T22:01:57.175153599Z INFO: on-disk MachineConfig config collection complete. 2026-07-06T22:01:57.487672353Z error: the server doesn't have a resource type "performanceprofile" 2026-07-06T22:01:57.493143287Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-06T22:01:57.614290132Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-06T22:01:57.627540060Z Wrote inspect data to must-gather. 2026-07-06T22:01:57.752368417Z Wrote inspect data to must-gather. 2026-07-06T22:01:58.061434862Z ERROR: No running kube-apiserver pods found 2026-07-06T22:01:58.083411817Z Wrote inspect data to must-gather. 2026-07-06T22:01:58.110266554Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T22:01:58.160771319Z INFO: Worker host service log collection to complete. 2026-07-06T22:01:58.165782881Z Wrote inspect data to must-gather. 2026-07-06T22:01:58.222508878Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-06T22:01:58.260890983Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-06T22:01:58.293605397Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-06T22:01:58.294783077Z No resources found 2026-07-06T22:01:58.347743000Z Wrote inspect data to must-gather. 2026-07-06T22:01:58.455676820Z Wrote inspect data to must-gather. 2026-07-06T22:01:58.464678947Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-06T22:01:58.542355597Z Wrote inspect data to must-gather. 2026-07-06T22:01:58.592702208Z INFO: 'previous-logs' folder not found on ip-10-0-136-12.ec2.internal, skipping... 2026-07-06T22:01:58.671226574Z tar: Removing leading `/' from member names 2026-07-06T22:01:58.677877777Z INFO: HAProxy config collection complete. 2026-07-06T22:01:58.751011172Z error: the server doesn't have a resource type "machineconfigs" 2026-07-06T22:01:58.757716255Z Wrote inspect data to must-gather. 2026-07-06T22:01:58.959190753Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-06T22:01:58.979664045Z INFO: 'previous-logs' folder not found on ip-10-0-140-54.ec2.internal, skipping... 2026-07-06T22:01:59.057364014Z INFO: Getting rules from prometheus-k8s-0 2026-07-06T22:01:59.064365755Z Gathering data for ns/openshift-monitoring... 2026-07-06T22:01:59.092298085Z Wrote inspect data to must-gather. 2026-07-06T22:01:59.257251374Z Wrote inspect data to must-gather. 2026-07-06T22:01:59.320492737Z INFO: 'previous-logs' folder not found on ip-10-0-142-144.ec2.internal, skipping... 2026-07-06T22:01:59.320591340Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-06T22:01:59.320643026Z INFO: Machine Config Daemon termination log collection complete. 2026-07-06T22:01:59.410489285Z Wrote inspect data to must-gather. 2026-07-06T22:01:59.437064035Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-06T22:01:59.534902034Z INFO: Getting status/config from prometheus-k8s-0 2026-07-06T22:01:59.675920984Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-06T22:01:59.703326794Z Wrote inspect data to must-gather. 2026-07-06T22:01:59.757360555Z INFO: INTERCONNECT MODE 2026-07-06T22:01:59.757391127Z INFO: Gathering ovn-kubernetes DBs 2026-07-06T22:01:59.934902516Z INFO: Gathering OVN_Northbound from ovnkube-node-fq68j... 2026-07-06T22:01:59.934939247Z Wrote inspect data to must-gather. 2026-07-06T22:01:59.935157795Z INFO: Gathering OVN_Northbound from ovnkube-node-g64lt... 2026-07-06T22:01:59.935424203Z INFO: Gathering OVN_Northbound from ovnkube-node-ntgf2... 2026-07-06T22:01:59.935708177Z INFO: Gathering OVN_Southbound from ovnkube-node-fq68j... 2026-07-06T22:01:59.936016300Z INFO: Gathering OVN_Southbound from ovnkube-node-g64lt... 2026-07-06T22:01:59.936276150Z INFO: Gathering OVN_Southbound from ovnkube-node-ntgf2... 2026-07-06T22:01:59.989329646Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-06T22:02:00.223287498Z tar: Removing leading `/' from member names 2026-07-06T22:02:00.248776364Z tar: Removing leading `/' from member names 2026-07-06T22:02:00.255668721Z tar: Removing leading `/' from member names 2026-07-06T22:02:00.269811376Z tar: Removing leading `/' from member names 2026-07-06T22:02:00.271743768Z tar: Removing leading `/' from member names 2026-07-06T22:02:00.292369709Z tar: Removing leading `/' from member names 2026-07-06T22:02:00.322418887Z Wrote inspect data to must-gather. 2026-07-06T22:02:00.394065926Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-06T22:02:00.520934465Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-06T22:02:00.606634702Z Wrote inspect data to must-gather. 2026-07-06T22:02:00.651452505Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-06T22:02:00.686878458Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-06T22:02:00.724859658Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:00.851598680Z Wrote inspect data to must-gather. 2026-07-06T22:02:00.903669418Z INFO: Waiting for network log collection to complete ... 2026-07-06T22:02:00.903729456Z INFO: Waiting for ovnk database copies to complete ... 2026-07-06T22:02:00.904473210Z INFO: Copying ovnk databases complete. 2026-07-06T22:02:00.906609507Z 3.9M must-gather/network_logs/ovnk_database_store 2026-07-06T22:02:00.909363992Z ovnk_database_store/ 2026-07-06T22:02:00.909403732Z ovnk_database_store/ovnkube-node-ntgf2_nbdb 2026-07-06T22:02:00.921554648Z ovnk_database_store/ovnkube-node-ntgf2_sbdb 2026-07-06T22:02:00.947746566Z ovnk_database_store/ovnkube-node-g64lt_sbdb 2026-07-06T22:02:00.964511538Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:00.964974073Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-06T22:02:00.967128691Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-06T22:02:00.972763212Z ovnk_database_store/ovnkube-node-fq68j_nbdb 2026-07-06T22:02:00.984418013Z ovnk_database_store/ovnkube-node-g64lt_nbdb 2026-07-06T22:02:00.996938815Z ovnk_database_store/ovnkube-node-fq68j_sbdb 2026-07-06T22:02:01.026549201Z INFO: Network log collection complete. 2026-07-06T22:02:01.114800364Z Wrote inspect data to must-gather. 2026-07-06T22:02:01.191746717Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:01.266575276Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T22:02:01.383264105Z INFO: Getting status from alertmanager-main-0 2026-07-06T22:02:01.434573255Z Wrote inspect data to must-gather. 2026-07-06T22:02:01.441941440Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:01.445006202Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-06T22:02:01.478320556Z Gathering data for ns/openshift-network-console... 2026-07-06T22:02:01.678439206Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:01.754542094Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-06T22:02:01.757583067Z INFO: Fallback to identify the container image from release info 2026-07-06T22:02:01.921653885Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:01.924082093Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-06T22:02:01.942178120Z Gathering data for ns/openshift-console-operator... 2026-07-06T22:02:02.148476020Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:02.335904703Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:02.339274755Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-06T22:02:02.427525879Z Gathering data for ns/openshift-console... 2026-07-06T22:02:02.511205622Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:02.652992352Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a 2026-07-06T22:02:02.731611251Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:02.734796974Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-06T22:02:02.789298358Z daemonset.apps/perf-node-gather-daemonset created 2026-07-06T22:02:02.943553476Z Waiting for performance profile collector pods to become ready: 1 2026-07-06T22:02:02.944081803Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:03.018369078Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-06T22:02:03.154026263Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:03.156704199Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-06T22:02:03.291314346Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-06T22:02:03.355032351Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:03.565036207Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:03.568130914Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-06T22:02:03.589420510Z Gathering data for ns/openshift-dns-operator... 2026-07-06T22:02:03.748124028Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:03.899991281Z Gathering data for ns/openshift-dns... 2026-07-06T22:02:03.947263605Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:03.949700761Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-06T22:02:04.078021642Z Waiting for performance profile collector pods to become ready: 2 2026-07-06T22:02:04.116144262Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:04.305257471Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:04.308674442Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-06T22:02:04.492568391Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:04.536415945Z Gathering data for ns/openshift-image-registry... 2026-07-06T22:02:04.670784209Z Wrote inspect data to /must-gather/istio. 2026-07-06T22:02:05.080601422Z Gathering data for ns/openshift-ingress-operator... 2026-07-06T22:02:05.229922302Z Waiting for performance profile collector pods to become ready: 3 2026-07-06T22:02:05.426524308Z Done executing Istio gather script 2026-07-06T22:02:05.445021477Z Gathering data for ns/openshift-ingress... 2026-07-06T22:02:05.764098837Z Gathering data for ns/openshift-ingress-canary... 2026-07-06T22:02:06.132916223Z Gathering data for ns/openshift-insights... 2026-07-06T22:02:06.350522680Z Waiting for performance profile collector pods to become ready: 4 2026-07-06T22:02:07.470185098Z Waiting for performance profile collector pods to become ready: 5 2026-07-06T22:02:08.294378600Z Gathering data for ns/kserve... 2026-07-06T22:02:08.295836422Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-06T22:02:08.610125580Z Waiting for performance profile collector pods to become ready: 6 2026-07-06T22:02:09.723435022Z Waiting for performance profile collector pods to become ready: 7 2026-07-06T22:02:09.872440137Z Gathering data for ns/openshift-config... 2026-07-06T22:02:10.166236391Z Gathering data for ns/openshift-config-managed... 2026-07-06T22:02:10.644625892Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-06T22:02:10.849978322Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-06T22:02:10.918416491Z Gathering data for ns/openshift-kube-apiserver... 2026-07-06T22:02:11.114077958Z Collecting performance related data for node ip-10-0-140-54.ec2.internal 2026-07-06T22:02:11.122360391Z Collecting performance related data for node ip-10-0-142-144.ec2.internal 2026-07-06T22:02:11.130494518Z Collecting performance related data for node ip-10-0-136-12.ec2.internal 2026-07-06T22:02:11.224063069Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-06T22:02:11.580968507Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-06T22:02:11.930192456Z Gathering data for ns/openshift-kube-scheduler... 2026-07-06T22:02:12.276261093Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-06T22:02:12.623120142Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-06T22:02:12.941050065Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-06T22:02:13.266239593Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-06T22:02:13.302145043Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-06T22:02:13.816164985Z Gathering data for ns/openshift-multus... 2026-07-06T22:02:15.125282207Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-06T22:02:16.375846596Z Gathering data for ns/openshift-host-network... 2026-07-06T22:02:16.693114803Z Gathering data for ns/openshift-network-diagnostics... 2026-07-06T22:02:17.217795555Z Gathering data for ns/openshift-network-node-identity... 2026-07-06T22:02:17.613500881Z Gathering data for ns/openshift-network-operator... 2026-07-06T22:02:17.967949086Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-06T22:02:18.232392933Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-06T22:02:18.307275952Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-07-06T22:02:18.709097185Z Gathering data for ns/openshift-apiserver-operator... 2026-07-06T22:02:19.012402040Z Gathering data for ns/openshift-apiserver... 2026-07-06T22:02:19.161542907Z Collecting kubelet logs for node ip-10-0-140-54.ec2.internal 2026-07-06T22:02:19.163176532Z Collecting kubelet logs for node ip-10-0-142-144.ec2.internal 2026-07-06T22:02:19.164917688Z Collecting kubelet logs for node ip-10-0-136-12.ec2.internal 2026-07-06T22:02:19.498143069Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-06T22:02:19.589001653Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-d2pl7 namespace 2026-07-06T22:02:19.593438649Z INFO: Node performance data collection complete. 2026-07-06T22:02:19.778970889Z Gathering data for ns/openshift-controller-manager... 2026-07-06T22:02:20.113081952Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-06T22:02:20.744644114Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-06T22:02:21.027092302Z Gathering data for ns/openshift-service-ca-operator... 2026-07-06T22:02:21.338209773Z Gathering data for ns/openshift-service-ca... 2026-07-06T22:02:21.679025718Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-06T22:02:22.581198736Z Wrote inspect data to must-gather. 2026-07-06T22:02:22.581241774Z error: inspection completed with the errors occurred while gathering data: 2026-07-06T22:02:22.581241774Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-06T22:02:22.708814005Z error: the server doesn't have a resource type "clusters" 2026-07-06T22:02:22.867932830Z Caches written to disk