2026-07-07T15:45:03.352670623Z [disk usage checker] Started
2026-07-07T15:45:03.356074378Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-07T15:45:03.550390002Z Gathering data for ns/openshift-cluster-version...
2026-07-07T15:45:03.697562273Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T15:45:03.798067788Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:45:03.884413423Z Gathering data for ns/default...
2026-07-07T15:45:04.218586886Z Gathering data for ns/openshift...
2026-07-07T15:45:04.681551189Z Gathering data for ns/kube-system...
2026-07-07T15:45:05.532988884Z Gathering data for ns/openshift-etcd...
2026-07-07T15:45:05.902362881Z Wrote inspect data to must-gather.
2026-07-07T15:45:05.902400815Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T15:45:05.902400815Z     namespaces "assisted-installer" not found
2026-07-07T15:45:06.136202890Z Waiting on subprocesses to finish execution.
2026-07-07T15:45:06.142071018Z Executing Istio gather script
2026-07-07T15:45:06.144338824Z INFO: Gathering HAProxy config files
2026-07-07T15:45:06.151181240Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T15:45:06.151239150Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T15:45:06.151881003Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T15:45:06.161778453Z INFO: Collecting host service logs for crio
2026-07-07T15:45:06.162181140Z INFO: Collecting host service logs for kubelet
2026-07-07T15:45:06.162639390Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T15:45:06.163085769Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T15:45:06.163448931Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T15:45:06.163811517Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T15:45:06.164161668Z INFO: Collecting host service logs for NetworkManager
2026-07-07T15:45:06.164503669Z INFO: Collecting host service logs for openvswitch
2026-07-07T15:45:06.164869563Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T15:45:06.165229083Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T15:45:06.165663940Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T15:45:06.166156173Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T15:45:06.174172928Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T15:45:06.680488975Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T15:45:06.680525986Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T15:45:07.035459239Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T15:45:07.081370473Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T15:45:07.152501822Z No resources found in openshift-etcd namespace.
2026-07-07T15:45:07.391041501Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T15:45:07.515860472Z No resources found
2026-07-07T15:45:07.690186378Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-n5ptf
2026-07-07T15:45:07.690186378Z insights-runtime-extractor-6mb4h
2026-07-07T15:45:07.690186378Z insights-runtime-extractor-6wjbk
2026-07-07T15:45:07.690186378Z insights-runtime-extractor-zwztf
2026-07-07T15:45:07.704872710Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T15:45:07.888757324Z Wrote inspect data to must-gather.
2026-07-07T15:45:08.007892883Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:45:08.070327520Z Wrote inspect data to must-gather.
2026-07-07T15:45:08.114092669Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T15:45:08.116782296Z INFO: Worker host service log collection to complete.
2026-07-07T15:45:08.257459587Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T15:45:08.302709009Z Wrote inspect data to must-gather.
2026-07-07T15:45:08.340904324Z Wrote inspect data to must-gather.
2026-07-07T15:45:08.443589695Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-07T15:45:08.448971735Z ERROR: No running kube-apiserver pods found
2026-07-07T15:45:08.541683731Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T15:45:08.541683731Z See 'oc rsync -h' for help and examples
2026-07-07T15:45:08.555515151Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T15:45:08.561468133Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T15:45:08.587314461Z No resources found
2026-07-07T15:45:08.617440244Z Wrote inspect data to must-gather.
2026-07-07T15:45:08.675555579Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T15:45:08.776969338Z INFO: 'previous-logs' folder not found on ip-10-0-128-215.ec2.internal, skipping...
2026-07-07T15:45:08.800097479Z Wrote inspect data to must-gather.
2026-07-07T15:45:08.839681364Z Gathering data for ns/openshift-monitoring...
2026-07-07T15:45:08.936996435Z Wrote inspect data to must-gather.
2026-07-07T15:45:09.037521103Z tar: Removing leading `/' from member names
2026-07-07T15:45:09.046658992Z INFO: HAProxy config collection complete.
2026-07-07T15:45:09.050548174Z Wrote inspect data to must-gather.
2026-07-07T15:45:09.157306442Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T15:45:09.177585737Z INFO: 'previous-logs' folder not found on ip-10-0-130-158.ec2.internal, skipping...
2026-07-07T15:45:09.207351007Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T15:45:09.380560430Z Wrote inspect data to must-gather.
2026-07-07T15:45:09.386470425Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T15:45:09.493965336Z INFO: 'previous-logs' folder not found on ip-10-0-134-178.ec2.internal, skipping...
2026-07-07T15:45:09.493997285Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T15:45:09.494021877Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T15:45:09.551954778Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T15:45:09.637551419Z Wrote inspect data to must-gather.
2026-07-07T15:45:09.660725743Z Wrote inspect data to must-gather.
2026-07-07T15:45:09.667063659Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T15:45:09.799012540Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T15:45:09.945463343Z INFO: INTERCONNECT MODE
2026-07-07T15:45:09.945488159Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T15:45:10.040930697Z Wrote inspect data to must-gather.
2026-07-07T15:45:10.095867953Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T15:45:10.118692080Z INFO: Gathering OVN_Northbound from ovnkube-node-ftfml...
2026-07-07T15:45:10.118934864Z INFO: Gathering OVN_Northbound from ovnkube-node-gpxk2...
2026-07-07T15:45:10.119194444Z INFO: Gathering OVN_Northbound from ovnkube-node-ssnrw...
2026-07-07T15:45:10.119449850Z INFO: Gathering OVN_Southbound from ovnkube-node-ftfml...
2026-07-07T15:45:10.119744151Z INFO: Gathering OVN_Southbound from ovnkube-node-gpxk2...
2026-07-07T15:45:10.120039845Z INFO: Gathering OVN_Southbound from ovnkube-node-ssnrw...
2026-07-07T15:45:10.404710618Z Wrote inspect data to must-gather.
2026-07-07T15:45:10.445266459Z tar: Removing leading `/' from member names
2026-07-07T15:45:10.445326884Z tar: Removing leading `/' from member names
2026-07-07T15:45:10.448067200Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T15:45:10.470331146Z tar: Removing leading `/' from member names
2026-07-07T15:45:10.523240938Z tar: Removing leading `/' from member names
2026-07-07T15:45:10.530114058Z tar: Removing leading `/' from member names
2026-07-07T15:45:10.542367700Z tar: Removing leading `/' from member names
2026-07-07T15:45:10.618191491Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T15:45:10.677475606Z Wrote inspect data to must-gather.
2026-07-07T15:45:10.721369586Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T15:45:10.940514892Z Wrote inspect data to must-gather.
2026-07-07T15:45:11.004596377Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T15:45:11.008507424Z INFO: Waiting for network log collection to complete ...
2026-07-07T15:45:11.008575968Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T15:45:11.009082366Z INFO: Copying ovnk databases complete.
2026-07-07T15:45:11.010486760Z 3.9M	must-gather/network_logs/ovnk_database_store
2026-07-07T15:45:11.012670788Z ovnk_database_store/
2026-07-07T15:45:11.012691437Z ovnk_database_store/ovnkube-node-gpxk2_sbdb
2026-07-07T15:45:11.034987590Z ovnk_database_store/ovnkube-node-gpxk2_nbdb
2026-07-07T15:45:11.044354694Z ovnk_database_store/ovnkube-node-ftfml_sbdb
2026-07-07T15:45:11.055025611Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T15:45:11.068626142Z ovnk_database_store/ovnkube-node-ftfml_nbdb
2026-07-07T15:45:11.081067059Z ovnk_database_store/ovnkube-node-ssnrw_sbdb
2026-07-07T15:45:11.111968132Z ovnk_database_store/ovnkube-node-ssnrw_nbdb
2026-07-07T15:45:11.128506249Z INFO: Network log collection complete.
2026-07-07T15:45:11.183162585Z W0707 15:45:11.183118    1477 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-07T15:45:11.188302867Z Wrote inspect data to must-gather.
2026-07-07T15:45:11.272686076Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:11.291713939Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T15:45:11.474126297Z INFO: Getting status from alertmanager-main-0
2026-07-07T15:45:11.499023407Z Gathering data for ns/openshift-network-console...
2026-07-07T15:45:11.501075705Z Wrote inspect data to must-gather.
2026-07-07T15:45:11.531546497Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:11.533940397Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T15:45:11.795629716Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:11.827971519Z Wrote inspect data to must-gather.
2026-07-07T15:45:12.013953154Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:12.016190456Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T15:45:12.017274790Z Gathering data for ns/openshift-console-operator...
2026-07-07T15:45:12.135179323Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T15:45:12.140520747Z INFO: Fallback to identify the container image from release info
2026-07-07T15:45:12.237758185Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:12.438612546Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:12.441167234Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T15:45:12.442205457Z Gathering data for ns/openshift-console...
2026-07-07T15:45:12.650124205Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:12.762618268Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T15:45:12.869226407Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:12.871650062Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-07T15:45:12.907319973Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T15:45:13.041471492Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T15:45:13.066867577Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T15:45:13.070303671Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:13.267510111Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:13.271136278Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-07T15:45:13.473347412Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-07T15:45:13.489915585Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:13.623709315Z Gathering data for ns/openshift-dns-operator...
2026-07-07T15:45:13.707842572Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:13.710485377Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-07T15:45:13.880398941Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:13.958118903Z Gathering data for ns/openshift-dns...
2026-07-07T15:45:14.075967157Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:14.078571510Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-07T15:45:14.197213127Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T15:45:14.262342505Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:14.460641245Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:14.463049434Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-07T15:45:14.651081072Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:14.655273511Z Gathering data for ns/openshift-image-registry...
2026-07-07T15:45:14.851502287Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:14.854841260Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T15:45:15.024227726Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:15.215101811Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T15:45:15.229437784Z Wrote inspect data to /must-gather/istio.
2026-07-07T15:45:15.346046503Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T15:45:15.621967573Z Gathering data for ns/openshift-ingress...
2026-07-07T15:45:15.940471905Z Done executing Istio gather script
2026-07-07T15:45:15.997610607Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T15:45:16.401912616Z Gathering data for ns/openshift-insights...
2026-07-07T15:45:16.481087884Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T15:45:17.593136401Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T15:45:18.477828096Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-07T15:45:18.717262148Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T15:45:18.846229098Z Gathering data for ns/kserve...
2026-07-07T15:45:19.851068688Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T15:45:20.498115747Z Gathering data for ns/openshift-config...
2026-07-07T15:45:20.774763277Z Gathering data for ns/openshift-config-managed...
2026-07-07T15:45:20.966089138Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T15:45:21.232199945Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T15:45:21.233349640Z Collecting performance related data for node ip-10-0-130-158.ec2.internal
2026-07-07T15:45:21.242888963Z Collecting performance related data for node ip-10-0-128-215.ec2.internal
2026-07-07T15:45:21.256202927Z Collecting performance related data for node ip-10-0-134-178.ec2.internal
2026-07-07T15:45:21.560522176Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T15:45:21.917303264Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T15:45:22.272911070Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T15:45:22.660406057Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T15:45:23.006245699Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T15:45:23.331034955Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T15:45:23.482657360Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-07T15:45:23.704679344Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T15:45:24.092208236Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T15:45:24.669598127Z Gathering data for ns/openshift-multus...
2026-07-07T15:45:26.180265908Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T15:45:27.571235701Z Gathering data for ns/openshift-host-network...
2026-07-07T15:45:27.906450098Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T15:45:28.488389973Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-07T15:45:28.511977380Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T15:45:28.993742489Z Gathering data for ns/openshift-network-operator...
2026-07-07T15:45:29.442686730Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T15:45:29.722345267Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T15:45:29.781367490Z Collecting kubelet logs for node ip-10-0-130-158.ec2.internal
2026-07-07T15:45:29.783051630Z Collecting kubelet logs for node ip-10-0-128-215.ec2.internal
2026-07-07T15:45:29.785187343Z Collecting kubelet logs for node ip-10-0-134-178.ec2.internal
2026-07-07T15:45:30.172502567Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-4xx6g namespace
2026-07-07T15:45:30.177783005Z INFO: Node performance data collection complete.
2026-07-07T15:45:30.290941915Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T15:45:30.648432007Z Gathering data for ns/openshift-apiserver...
2026-07-07T15:45:31.037425000Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T15:45:31.314393593Z Gathering data for ns/openshift-controller-manager...
2026-07-07T15:45:31.590326280Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T15:45:32.238512099Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T15:45:32.531991712Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T15:45:32.845335060Z Gathering data for ns/openshift-service-ca...
2026-07-07T15:45:33.207457665Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T15:45:33.494039870Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-07T15:45:34.162144389Z Wrote inspect data to must-gather.
2026-07-07T15:45:34.162198395Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T15:45:34.162198395Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T15:45:34.293215908Z error: the server doesn't have a resource type "clusters"
2026-07-07T15:45:34.606487313Z Caches written to disk