2026-07-02T21:45:14.998178519Z [disk usage checker] Started
2026-07-02T21:45:15.001643079Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-02T21:45:15.300572310Z Gathering data for ns/openshift-cluster-version...
2026-07-02T21:45:15.446310558Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T21:45:15.538934635Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T21:45:15.627345704Z Gathering data for ns/default...
2026-07-02T21:45:15.971290358Z Gathering data for ns/openshift...
2026-07-02T21:45:16.399518177Z Gathering data for ns/kube-system...
2026-07-02T21:45:17.142033314Z Gathering data for ns/openshift-etcd...
2026-07-02T21:45:17.461958728Z Wrote inspect data to must-gather.
2026-07-02T21:45:17.461992722Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T21:45:17.461992722Z     namespaces "assisted-installer" not found
2026-07-02T21:45:18.189087350Z Waiting on subprocesses to finish execution.
2026-07-02T21:45:18.196862853Z INFO: Gathering HAProxy config files
2026-07-02T21:45:18.199065075Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T21:45:18.202409189Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T21:45:18.209663417Z INFO: Collecting host service logs for crio
2026-07-02T21:45:18.210831704Z INFO: Collecting host service logs for kubelet
2026-07-02T21:45:18.211223774Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T21:45:18.211579661Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T21:45:18.211951797Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T21:45:18.212324229Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T21:45:18.212699321Z INFO: Collecting host service logs for NetworkManager
2026-07-02T21:45:18.213059098Z INFO: Collecting host service logs for openvswitch
2026-07-02T21:45:18.213416079Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T21:45:18.213784999Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T21:45:18.214139006Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T21:45:18.214568682Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T21:45:18.216509703Z Executing Istio gather script
2026-07-02T21:45:18.225824821Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T21:45:18.567452112Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T21:45:18.714813465Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T21:45:18.974714319Z No resources found in openshift-etcd namespace.
2026-07-02T21:45:19.033863390Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T21:45:19.270965045Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-5qf9s
2026-07-02T21:45:19.271035079Z insights-runtime-extractor-dldh9
2026-07-02T21:45:19.271055878Z insights-runtime-extractor-fkt9s
2026-07-02T21:45:19.271075284Z insights-runtime-extractor-kw9lq
2026-07-02T21:45:19.484785133Z No resources found
2026-07-02T21:45:19.515703718Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T21:45:19.515823908Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T21:45:19.517975058Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T21:45:19.529388394Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T21:45:19.529388394Z See 'oc rsync -h' for help and examples
2026-07-02T21:45:19.715200181Z ERROR: No running kube-apiserver pods found
2026-07-02T21:45:19.780904357Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T21:45:19.911854230Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T21:45:19.931269926Z Wrote inspect data to must-gather.
2026-07-02T21:45:19.970089481Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T21:45:20.057456800Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T21:45:20.114437847Z Gathering data for ns/openshift-monitoring...
2026-07-02T21:45:20.114437847Z INFO: Worker host service log collection to complete.
2026-07-02T21:45:20.169932852Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-02T21:45:20.477958435Z Wrote inspect data to must-gather.
2026-07-02T21:45:20.533921657Z Wrote inspect data to must-gather.
2026-07-02T21:45:20.535239132Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T21:45:20.586153591Z INFO: 'previous-logs' folder not found on ip-10-0-134-124.ec2.internal, skipping...
2026-07-02T21:45:20.661136110Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T21:45:20.769001084Z Wrote inspect data to must-gather.
2026-07-02T21:45:20.800027973Z tar: Removing leading `/' from member names
2026-07-02T21:45:20.809407257Z INFO: HAProxy config collection complete.
2026-07-02T21:45:20.880102061Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T21:45:20.884229448Z No resources found
2026-07-02T21:45:20.978077389Z Wrote inspect data to must-gather.
2026-07-02T21:45:21.006256126Z Wrote inspect data to must-gather.
2026-07-02T21:45:21.035855678Z Wrote inspect data to must-gather.
2026-07-02T21:45:21.103451596Z INFO: 'previous-logs' folder not found on ip-10-0-140-1.ec2.internal, skipping...
2026-07-02T21:45:21.190166425Z Wrote inspect data to must-gather.
2026-07-02T21:45:21.246150332Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T21:45:21.329681540Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T21:45:21.406855582Z INFO: 'previous-logs' folder not found on ip-10-0-141-123.ec2.internal, skipping...
2026-07-02T21:45:21.406855582Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T21:45:21.406898576Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T21:45:21.445939255Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T21:45:21.508036522Z Wrote inspect data to must-gather.
2026-07-02T21:45:21.696410143Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T21:45:21.713596860Z Wrote inspect data to must-gather.
2026-07-02T21:45:21.782707993Z Wrote inspect data to must-gather.
2026-07-02T21:45:21.809723093Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T21:45:21.908122356Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T21:45:21.974052189Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T21:45:22.162491333Z INFO: INTERCONNECT MODE
2026-07-02T21:45:22.162535540Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T21:45:22.169049494Z Wrote inspect data to must-gather.
2026-07-02T21:45:22.297686570Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T21:45:22.378482046Z INFO: Gathering OVN_Northbound from ovnkube-node-28cz9...
2026-07-02T21:45:22.378729599Z INFO: Gathering OVN_Northbound from ovnkube-node-jhw2p...
2026-07-02T21:45:22.378982444Z INFO: Gathering OVN_Northbound from ovnkube-node-mvqc5...
2026-07-02T21:45:22.379266726Z INFO: Gathering OVN_Southbound from ovnkube-node-28cz9...
2026-07-02T21:45:22.379560921Z INFO: Gathering OVN_Southbound from ovnkube-node-jhw2p...
2026-07-02T21:45:22.379893675Z INFO: Gathering OVN_Southbound from ovnkube-node-mvqc5...
2026-07-02T21:45:22.587762573Z Wrote inspect data to must-gather.
2026-07-02T21:45:22.696085977Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T21:45:22.748088970Z tar: Removing leading `/' from member names
2026-07-02T21:45:22.760301798Z tar: Removing leading `/' from member names
2026-07-02T21:45:22.760473532Z tar: Removing leading `/' from member names
2026-07-02T21:45:22.770829738Z tar: Removing leading `/' from member names
2026-07-02T21:45:22.788511738Z tar: Removing leading `/' from member names
2026-07-02T21:45:22.790038071Z tar: Removing leading `/' from member names
2026-07-02T21:45:22.829370847Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T21:45:22.982737606Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T21:45:23.007811955Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T21:45:23.053345217Z Wrote inspect data to must-gather.
2026-07-02T21:45:23.070340222Z Gathering data for ns/openshift-network-console...
2026-07-02T21:45:23.278019135Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:23.287585472Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T21:45:23.316942662Z INFO: Waiting for network log collection to complete ...
2026-07-02T21:45:23.317064385Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T21:45:23.317479877Z INFO: Copying ovnk databases complete.
2026-07-02T21:45:23.319462373Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-02T21:45:23.321911834Z ovnk_database_store/
2026-07-02T21:45:23.321962492Z ovnk_database_store/ovnkube-node-28cz9_sbdb
2026-07-02T21:45:23.347417503Z ovnk_database_store/ovnkube-node-mvqc5_nbdb
2026-07-02T21:45:23.358977506Z ovnk_database_store/ovnkube-node-jhw2p_sbdb
2026-07-02T21:45:23.378972438Z ovnk_database_store/ovnkube-node-28cz9_nbdb
2026-07-02T21:45:23.388457294Z ovnk_database_store/ovnkube-node-jhw2p_nbdb
2026-07-02T21:45:23.398569544Z ovnk_database_store/ovnkube-node-mvqc5_sbdb
2026-07-02T21:45:23.426752965Z INFO: Network log collection complete.
2026-07-02T21:45:23.428130107Z Wrote inspect data to must-gather.
2026-07-02T21:45:23.545678974Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:23.549280036Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T21:45:23.628178273Z Gathering data for ns/openshift-console-operator...
2026-07-02T21:45:23.755130365Z INFO: Getting status from alertmanager-main-0
2026-07-02T21:45:23.787016159Z Wrote inspect data to must-gather.
2026-07-02T21:45:23.787614071Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:24.031253590Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:24.036158070Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T21:45:24.099247846Z Gathering data for ns/openshift-console...
2026-07-02T21:45:24.126171922Z Wrote inspect data to must-gather.
2026-07-02T21:45:24.267666979Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:24.447535082Z Wrote inspect data to must-gather.
2026-07-02T21:45:24.497165082Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:24.501734238Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T21:45:24.661107859Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T21:45:24.760330985Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:24.768730478Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T21:45:24.772029203Z INFO: Fallback to identify the container image from release info
2026-07-02T21:45:24.980429075Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:24.984561874Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-02T21:45:25.168779045Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:25.196757713Z Gathering data for ns/openshift-dns-operator...
2026-07-02T21:45:25.233760776Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-02T21:45:25.399162152Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:25.402597864Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-02T21:45:25.462752898Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-02T21:45:25.555020066Z Gathering data for ns/openshift-dns...
2026-07-02T21:45:25.614009621Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T21:45:25.629507404Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:25.750600750Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T21:45:25.827805279Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:25.832706112Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-02T21:45:26.023756543Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:26.255527137Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:26.259792850Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-02T21:45:26.275028182Z Gathering data for ns/openshift-image-registry...
2026-07-02T21:45:26.440819168Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:26.631769806Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:26.634794901Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-02T21:45:26.819852453Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:26.905570889Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T21:45:26.914584463Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T21:45:27.045574585Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:27.048595870Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T21:45:27.237434975Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:27.298979068Z Gathering data for ns/openshift-ingress...
2026-07-02T21:45:27.456235058Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:45:27.671261731Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T21:45:28.066128695Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T21:45:28.181225761Z Gathering data for ns/openshift-insights...
2026-07-02T21:45:28.208214452Z Done executing Istio gather script
2026-07-02T21:45:29.184499870Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T21:45:30.240827694Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-02T21:45:30.308411121Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T21:45:30.388361073Z Gathering data for ns/kserve...
2026-07-02T21:45:31.437600480Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T21:45:32.063743014Z Gathering data for ns/openshift-config...
2026-07-02T21:45:32.326459419Z Gathering data for ns/openshift-config-managed...
2026-07-02T21:45:32.556705098Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T21:45:32.779926975Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T21:45:32.839779594Z Collecting performance related data for node ip-10-0-134-124.ec2.internal
2026-07-02T21:45:32.848262389Z Collecting performance related data for node ip-10-0-141-123.ec2.internal
2026-07-02T21:45:32.862331797Z Collecting performance related data for node ip-10-0-140-1.ec2.internal
2026-07-02T21:45:33.107760471Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T21:45:33.429916324Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T21:45:33.815353210Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T21:45:34.204167210Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T21:45:34.531852913Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T21:45:34.875927563Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T21:45:35.208681762Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T21:45:35.248634750Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-02T21:45:35.554771190Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T21:45:36.163881000Z Gathering data for ns/openshift-multus...
2026-07-02T21:45:37.586567949Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T21:45:38.917932304Z Gathering data for ns/openshift-host-network...
2026-07-02T21:45:39.226775235Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T21:45:39.821456936Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T21:45:40.245069202Z Gathering data for ns/openshift-network-operator...
2026-07-02T21:45:40.253940478Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-02T21:45:40.636153823Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T21:45:40.897105708Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T21:45:41.416498443Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T21:45:41.450208562Z Collecting kubelet logs for node ip-10-0-134-124.ec2.internal
2026-07-02T21:45:41.452435088Z Collecting kubelet logs for node ip-10-0-141-123.ec2.internal
2026-07-02T21:45:41.454597168Z Collecting kubelet logs for node ip-10-0-140-1.ec2.internal
2026-07-02T21:45:41.773027642Z Gathering data for ns/openshift-apiserver...
2026-07-02T21:45:41.863888283Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-p2hh8 namespace
2026-07-02T21:45:41.869190554Z INFO: Node performance data collection complete.
2026-07-02T21:45:42.206037906Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T21:45:42.507150751Z Gathering data for ns/openshift-controller-manager...
2026-07-02T21:45:42.772926129Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T21:45:43.401199893Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T21:45:43.686776118Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T21:45:43.995863027Z Gathering data for ns/openshift-service-ca...
2026-07-02T21:45:44.371707231Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T21:45:45.260433051Z [disk usage checker] Volume usage percentage: current = 25 ; allowed = 70
2026-07-02T21:45:45.296955705Z Wrote inspect data to must-gather.
2026-07-02T21:45:45.296996105Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T21:45:45.296996105Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T21:45:45.432385863Z error: the server doesn't have a resource type "clusters"
2026-07-02T21:45:45.562797436Z Caches written to disk