2026-09-18T20:06:07.737755186Z [disk usage checker] Started
2026-09-18T20:06:07.741282938Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-09-18T20:06:07.937728975Z Gathering data for ns/openshift-cluster-version...
2026-09-18T20:06:08.092626997Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-18T20:06:08.227845178Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-18T20:06:08.314344374Z Gathering data for ns/default...
2026-09-18T20:06:08.661822440Z Gathering data for ns/openshift...
2026-09-18T20:06:09.139289086Z Gathering data for ns/kube-system...
2026-09-18T20:06:09.950099278Z Gathering data for ns/openshift-etcd...
2026-09-18T20:06:10.319157913Z Wrote inspect data to must-gather.
2026-09-18T20:06:10.319199059Z error: inspection completed with the errors occurred while gathering data:
2026-09-18T20:06:10.319199059Z     namespaces "assisted-installer" not found
2026-09-18T20:06:10.495042789Z Waiting on subprocesses to finish execution.
2026-09-18T20:06:10.505956688Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-18T20:06:10.514653183Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-18T20:06:10.514996816Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-18T20:06:10.536931403Z INFO: Gathering HAProxy config files
2026-09-18T20:06:10.550535140Z INFO: Collecting host service logs for crio
2026-09-18T20:06:10.550915598Z INFO: Collecting host service logs for kubelet
2026-09-18T20:06:10.551277325Z INFO: Collecting host service logs for rpm-ostreed
2026-09-18T20:06:10.551687176Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-18T20:06:10.552034330Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-18T20:06:10.552356721Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-18T20:06:10.552705612Z INFO: Collecting host service logs for NetworkManager
2026-09-18T20:06:10.553054911Z INFO: Collecting host service logs for openvswitch
2026-09-18T20:06:10.553377504Z INFO: Collecting host service logs for ovs-configuration
2026-09-18T20:06:10.553743181Z INFO: Collecting host service logs for ovsdb-server
2026-09-18T20:06:10.554077843Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-18T20:06:10.554477198Z INFO: Waiting for worker host service log collection to complete ...
2026-09-18T20:06:10.562587174Z Executing Istio gather script
2026-09-18T20:06:10.586512251Z INFO: Waiting for node performance related collection to complete ...
2026-09-18T20:06:11.160883624Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-18T20:06:11.160971362Z INFO: on-disk MachineConfig config collection complete.
2026-09-18T20:06:11.335068262Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-18T20:06:11.385130404Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-18T20:06:11.490242403Z INFO: Collecting Insights Archives from insights-operator-6f9c644dd9-chdrd
2026-09-18T20:06:11.490289731Z insights-runtime-extractor-rbnz8
2026-09-18T20:06:11.490309835Z insights-runtime-extractor-shm2b
2026-09-18T20:06:11.490329705Z insights-runtime-extractor-wqxts
2026-09-18T20:06:11.535390774Z error: the server doesn't have a resource type "performanceprofile"
2026-09-18T20:06:11.562897149Z INFO: "metallb-operator" not detected. Skipping.
2026-09-18T20:06:11.623213796Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-18T20:06:11.635245518Z No resources found in openshift-etcd namespace.
2026-09-18T20:06:11.759986624Z No resources found
2026-09-18T20:06:11.821232905Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-18T20:06:11.967751515Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-18T20:06:11.967751515Z See 'oc rsync -h' for help and examples
2026-09-18T20:06:12.037248751Z INFO: Worker host service log collection to complete.
2026-09-18T20:06:12.093383872Z Wrote inspect data to must-gather.
2026-09-18T20:06:12.147910758Z Wrote inspect data to must-gather.
2026-09-18T20:06:12.185346843Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-18T20:06:12.209062592Z ERROR: No running kube-apiserver pods found
2026-09-18T20:06:12.544345780Z INFO: 'previous-logs' folder not found on ip-10-0-129-212.ec2.internal, skipping...
2026-09-18T20:06:12.561584616Z Wrote inspect data to must-gather.
2026-09-18T20:06:12.591546019Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-18T20:06:12.920884656Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-09-18T20:06:12.977219713Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-18T20:06:12.995738030Z No resources found
2026-09-18T20:06:13.014866051Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-18T20:06:13.130557755Z Wrote inspect data to must-gather.
2026-09-18T20:06:13.283960109Z Wrote inspect data to must-gather.
2026-09-18T20:06:13.318265815Z Wrote inspect data to must-gather.
2026-09-18T20:06:13.337621431Z error: the server doesn't have a resource type "machineconfigs"
2026-09-18T20:06:13.355232105Z Wrote inspect data to must-gather.
2026-09-18T20:06:13.356583809Z INFO: 'previous-logs' folder not found on ip-10-0-140-94.ec2.internal, skipping...
2026-09-18T20:06:13.429943751Z Gathering data for ns/openshift-monitoring...
2026-09-18T20:06:13.445097488Z INFO: Getting rules from prometheus-k8s-0
2026-09-18T20:06:13.573109583Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-18T20:06:13.588024115Z tar: Removing leading `/' from member names
2026-09-18T20:06:13.593737514Z INFO: HAProxy config collection complete.
2026-09-18T20:06:13.615549286Z Wrote inspect data to must-gather.
2026-09-18T20:06:13.688807405Z INFO: 'previous-logs' folder not found on ip-10-0-143-37.ec2.internal, skipping...
2026-09-18T20:06:13.688843947Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-18T20:06:13.688872581Z INFO: Machine Config Daemon termination log collection complete.
2026-09-18T20:06:13.820372741Z Wrote inspect data to must-gather.
2026-09-18T20:06:13.911171826Z Wrote inspect data to must-gather.
2026-09-18T20:06:13.927656844Z INFO: Getting status/config from prometheus-k8s-0
2026-09-18T20:06:13.993167097Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-18T20:06:14.214078527Z Wrote inspect data to must-gather.
2026-09-18T20:06:14.231690432Z Wrote inspect data to must-gather.
2026-09-18T20:06:14.282629897Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-18T20:06:14.391302336Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-18T20:06:14.482450551Z Wrote inspect data to must-gather.
2026-09-18T20:06:14.540566182Z INFO: INTERCONNECT MODE
2026-09-18T20:06:14.540612204Z INFO: Gathering ovn-kubernetes DBs
2026-09-18T20:06:14.659629729Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-18T20:06:14.704141031Z INFO: Gathering OVN_Northbound from ovnkube-node-bqqtf...
2026-09-18T20:06:14.704361693Z INFO: Gathering OVN_Northbound from ovnkube-node-j5x2c...
2026-09-18T20:06:14.704610265Z INFO: Gathering OVN_Northbound from ovnkube-node-rx96p...
2026-09-18T20:06:14.705026530Z INFO: Gathering OVN_Southbound from ovnkube-node-bqqtf...
2026-09-18T20:06:14.705337389Z INFO: Gathering OVN_Southbound from ovnkube-node-j5x2c...
2026-09-18T20:06:14.705621565Z INFO: Gathering OVN_Southbound from ovnkube-node-rx96p...
2026-09-18T20:06:14.712453337Z Wrote inspect data to must-gather.
2026-09-18T20:06:14.998185834Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-18T20:06:15.028115672Z tar: Removing leading `/' from member names
2026-09-18T20:06:15.038729054Z tar: Removing leading `/' from member names
2026-09-18T20:06:15.056750992Z tar: Removing leading `/' from member names
2026-09-18T20:06:15.057284539Z tar: Removing leading `/' from member names
2026-09-18T20:06:15.067017536Z Wrote inspect data to must-gather.
2026-09-18T20:06:15.083785818Z tar: Removing leading `/' from member names
2026-09-18T20:06:15.086683995Z tar: Removing leading `/' from member names
2026-09-18T20:06:15.306357753Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-18T20:06:15.341234884Z Wrote inspect data to must-gather.
2026-09-18T20:06:15.437341311Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-18T20:06:15.463438469Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-18T20:06:15.586834446Z Wrote inspect data to must-gather.
2026-09-18T20:06:15.649520501Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:15.707747375Z INFO: Waiting for network log collection to complete ...
2026-09-18T20:06:15.707868923Z INFO: Waiting for ovnk database copies to complete ...
2026-09-18T20:06:15.708393448Z INFO: Copying ovnk databases complete.
2026-09-18T20:06:15.710301886Z 3.9M	must-gather/network_logs/ovnk_database_store
2026-09-18T20:06:15.712752564Z ovnk_database_store/
2026-09-18T20:06:15.712798237Z ovnk_database_store/ovnkube-node-bqqtf_sbdb
2026-09-18T20:06:15.737316936Z ovnk_database_store/ovnkube-node-j5x2c_sbdb
2026-09-18T20:06:15.763404923Z ovnk_database_store/ovnkube-node-bqqtf_nbdb
2026-09-18T20:06:15.772708377Z INFO: Getting status from alertmanager-main-0
2026-09-18T20:06:15.772735587Z ovnk_database_store/ovnkube-node-rx96p_nbdb
2026-09-18T20:06:15.784859234Z ovnk_database_store/ovnkube-node-rx96p_sbdb
2026-09-18T20:06:15.816487227Z ovnk_database_store/ovnkube-node-j5x2c_nbdb
2026-09-18T20:06:15.829085772Z INFO: Network log collection complete.
2026-09-18T20:06:15.856572461Z Wrote inspect data to must-gather.
2026-09-18T20:06:15.912931726Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:15.916847562Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-18T20:06:16.120843019Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:16.149827922Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-18T20:06:16.155153755Z INFO: Fallback to identify the container image from release info
2026-09-18T20:06:16.230065037Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-18T20:06:16.348106374Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:16.350414049Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-18T20:06:16.407316662Z Gathering data for ns/openshift-network-console...
2026-09-18T20:06:16.553321256Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:16.714753490Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0720c8b078cc0cebb01b4c5cde21180aee7c2bb1a0d322b21890c0d3707406ea
2026-09-18T20:06:16.768536957Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:16.772284251Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-18T20:06:16.878577366Z daemonset.apps/perf-node-gather-daemonset created
2026-09-18T20:06:17.021451294Z Waiting for performance profile collector pods to become ready: 1
2026-09-18T20:06:17.056033670Z Gathering data for ns/openshift-console-operator...
2026-09-18T20:06:17.089019873Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:17.309209504Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:17.312158046Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-09-18T20:06:17.487280084Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:17.708719609Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:17.711170632Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-09-18T20:06:17.718400052Z Gathering data for ns/openshift-console...
2026-09-18T20:06:17.890204105Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:17.982884711Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-09-18T20:06:18.071016599Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:18.073844684Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-18T20:06:18.176164249Z Waiting for performance profile collector pods to become ready: 2
2026-09-18T20:06:18.261971779Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:18.419006326Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-18T20:06:18.487304454Z Wrote inspect data to /must-gather/istio.
2026-09-18T20:06:19.125756268Z Gathering data for ns/openshift-dns-operator...
2026-09-18T20:06:19.211438416Z Done executing Istio gather script
2026-09-18T20:06:19.311414170Z Waiting for performance profile collector pods to become ready: 3
2026-09-18T20:06:19.499195145Z Gathering data for ns/openshift-dns...
2026-09-18T20:06:20.225829520Z Gathering data for ns/openshift-image-registry...
2026-09-18T20:06:20.451790993Z Waiting for performance profile collector pods to become ready: 4
2026-09-18T20:06:20.752358002Z Gathering data for ns/openshift-ingress-operator...
2026-09-18T20:06:21.095336500Z Gathering data for ns/openshift-ingress...
2026-09-18T20:06:21.440681923Z Gathering data for ns/openshift-ingress-canary...
2026-09-18T20:06:21.571882694Z Waiting for performance profile collector pods to become ready: 5
2026-09-18T20:06:21.870936489Z Gathering data for ns/openshift-insights...
2026-09-18T20:06:22.691345097Z Waiting for performance profile collector pods to become ready: 6
2026-09-18T20:06:22.987847509Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-09-18T20:06:23.823835674Z Waiting for performance profile collector pods to become ready: 7
2026-09-18T20:06:24.403674565Z Gathering data for ns/kserve...
2026-09-18T20:06:24.936579854Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-18T20:06:25.195910419Z Collecting performance related data for node ip-10-0-129-212.ec2.internal
2026-09-18T20:06:25.203506317Z Collecting performance related data for node ip-10-0-140-94.ec2.internal
2026-09-18T20:06:25.212510583Z Collecting performance related data for node ip-10-0-143-37.ec2.internal
2026-09-18T20:06:26.378276989Z Gathering data for ns/openshift-config...
2026-09-18T20:06:26.812769022Z Gathering data for ns/openshift-config-managed...
2026-09-18T20:06:27.356387070Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-18T20:06:27.690054290Z Gathering data for ns/openshift-kube-apiserver...
2026-09-18T20:06:27.994622137Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-09-18T20:06:28.034742357Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-18T20:06:28.431310098Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-18T20:06:28.787057978Z Gathering data for ns/openshift-kube-scheduler...
2026-09-18T20:06:29.107020916Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-18T20:06:29.491984345Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-18T20:06:29.914406938Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-18T20:06:30.319441777Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-18T20:06:30.929329755Z Gathering data for ns/openshift-multus...
2026-09-18T20:06:32.829587178Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-18T20:06:32.999314121Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-09-18T20:06:33.311044045Z Collecting kubelet logs for node ip-10-0-129-212.ec2.internal
2026-09-18T20:06:33.312981295Z Collecting kubelet logs for node ip-10-0-140-94.ec2.internal
2026-09-18T20:06:33.314812358Z Collecting kubelet logs for node ip-10-0-143-37.ec2.internal
2026-09-18T20:06:33.687412494Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-7jg6d namespace
2026-09-18T20:06:33.693847042Z INFO: Node performance data collection complete.
2026-09-18T20:06:34.501926346Z Gathering data for ns/openshift-host-network...
2026-09-18T20:06:34.864022163Z Gathering data for ns/openshift-network-diagnostics...
2026-09-18T20:06:35.475908835Z Gathering data for ns/openshift-network-node-identity...
2026-09-18T20:06:36.067873760Z Gathering data for ns/openshift-network-operator...
2026-09-18T20:06:36.641174733Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-18T20:06:37.009869139Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-18T20:06:37.687760425Z Gathering data for ns/openshift-apiserver-operator...
2026-09-18T20:06:38.004064581Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-09-18T20:06:38.005157509Z Gathering data for ns/openshift-apiserver...
2026-09-18T20:06:38.486660083Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-18T20:06:38.827873653Z Gathering data for ns/openshift-controller-manager...
2026-09-18T20:06:39.132580987Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-18T20:06:39.806432871Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-18T20:06:40.182796249Z Gathering data for ns/openshift-service-ca-operator...
2026-09-18T20:06:40.577462890Z Gathering data for ns/openshift-service-ca...
2026-09-18T20:06:41.014286352Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-18T20:06:42.103901478Z Wrote inspect data to must-gather.
2026-09-18T20:06:42.103942457Z error: inspection completed with the errors occurred while gathering data:
2026-09-18T20:06:42.103942457Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-18T20:06:42.239769701Z error: the server doesn't have a resource type "clusters"
2026-09-18T20:06:42.307221956Z Caches written to disk