2026-07-07T14:36:40.254100036Z [disk usage checker] Started
2026-07-07T14:36:40.257692272Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:36:40.476621021Z Gathering data for ns/openshift-cluster-version...
2026-07-07T14:36:40.646411183Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T14:36:40.770602674Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:36:40.871685450Z Gathering data for ns/default...
2026-07-07T14:36:41.331184251Z Gathering data for ns/openshift...
2026-07-07T14:36:41.904008292Z Gathering data for ns/kube-system...
2026-07-07T14:36:42.846818574Z Gathering data for ns/openshift-etcd...
2026-07-07T14:36:43.146911110Z Waiting on subprocesses to finish execution.
2026-07-07T14:36:43.153255944Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T14:36:43.158918142Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T14:36:43.160074587Z Executing Istio gather script
2026-07-07T14:36:43.175604296Z INFO: Collecting host service logs for crio
2026-07-07T14:36:43.176002730Z INFO: Collecting host service logs for kubelet
2026-07-07T14:36:43.176338152Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T14:36:43.176686672Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T14:36:43.177035059Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T14:36:43.177389380Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T14:36:43.177849384Z INFO: Collecting host service logs for NetworkManager
2026-07-07T14:36:43.178178802Z INFO: Collecting host service logs for openvswitch
2026-07-07T14:36:43.178499669Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T14:36:43.178867127Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T14:36:43.179260770Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T14:36:43.179792893Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T14:36:43.202431568Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T14:36:43.209370452Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T14:36:43.229945443Z INFO: Gathering HAProxy config files
2026-07-07T14:36:43.802304686Z Wrote inspect data to must-gather.
2026-07-07T14:36:43.871309107Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T14:36:43.871309107Z     namespaces "assisted-installer" not found
2026-07-07T14:36:44.021727536Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T14:36:44.200753315Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T14:36:44.311361358Z Wrote inspect data to must-gather.
2026-07-07T14:36:44.317537455Z No resources found
2026-07-07T14:36:44.339530727Z No resources found in openshift-etcd namespace.
2026-07-07T14:36:44.391881836Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T14:36:44.548441937Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T14:36:44.548525234Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T14:36:44.576976602Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T14:36:44.629741166Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-k8t8j
2026-07-07T14:36:44.629741166Z insights-runtime-extractor-45gjt
2026-07-07T14:36:44.629741166Z insights-runtime-extractor-jd5qx
2026-07-07T14:36:44.629741166Z insights-runtime-extractor-kh9z5
2026-07-07T14:36:44.765488684Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T14:36:44.836986060Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T14:36:44.899926039Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T14:36:44.918691125Z INFO: Worker host service log collection to complete.
2026-07-07T14:36:45.204872578Z ERROR: No running kube-apiserver pods found
2026-07-07T14:36:45.286627302Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T14:36:45.286627302Z See 'oc rsync -h' for help and examples
2026-07-07T14:36:45.401953720Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:36:45.456805580Z Wrote inspect data to must-gather.
2026-07-07T14:36:45.472256742Z Wrote inspect data to must-gather.
2026-07-07T14:36:45.580996040Z Wrote inspect data to must-gather.
2026-07-07T14:36:45.887594058Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T14:36:45.909262408Z Wrote inspect data to must-gather.
2026-07-07T14:36:45.912249113Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T14:36:45.919178486Z tar: Removing leading `/' from member names
2026-07-07T14:36:45.922155864Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:36:45.927380449Z INFO: HAProxy config collection complete.
2026-07-07T14:36:45.934842344Z INFO: 'previous-logs' folder not found on ip-10-0-130-166.ec2.internal, skipping...
2026-07-07T14:36:45.947015972Z No resources found
2026-07-07T14:36:46.035960069Z Wrote inspect data to must-gather.
2026-07-07T14:36:46.049809330Z Wrote inspect data to must-gather.
2026-07-07T14:36:46.139214250Z Wrote inspect data to must-gather.
2026-07-07T14:36:46.253128001Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T14:36:46.315711980Z INFO: 'previous-logs' folder not found on ip-10-0-135-29.ec2.internal, skipping...
2026-07-07T14:36:46.398111860Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T14:36:46.453889772Z Wrote inspect data to must-gather.
2026-07-07T14:36:46.473149700Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T14:36:46.507027552Z Gathering data for ns/openshift-monitoring...
2026-07-07T14:36:46.521017091Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T14:36:46.624264421Z INFO: 'previous-logs' folder not found on ip-10-0-138-42.ec2.internal, skipping...
2026-07-07T14:36:46.624322514Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T14:36:46.624392723Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T14:36:46.748746454Z Wrote inspect data to must-gather.
2026-07-07T14:36:46.767130966Z Wrote inspect data to must-gather.
2026-07-07T14:36:46.923660519Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T14:36:47.041646378Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T14:36:47.096933455Z INFO: INTERCONNECT MODE
2026-07-07T14:36:47.097015208Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T14:36:47.174345472Z Wrote inspect data to must-gather.
2026-07-07T14:36:47.270972261Z INFO: Gathering OVN_Northbound from ovnkube-node-5jd26...
2026-07-07T14:36:47.271261781Z INFO: Gathering OVN_Northbound from ovnkube-node-ddpch...
2026-07-07T14:36:47.271490019Z INFO: Gathering OVN_Northbound from ovnkube-node-xzqcm...
2026-07-07T14:36:47.271797469Z INFO: Gathering OVN_Southbound from ovnkube-node-5jd26...
2026-07-07T14:36:47.272034583Z INFO: Gathering OVN_Southbound from ovnkube-node-ddpch...
2026-07-07T14:36:47.272263811Z INFO: Gathering OVN_Southbound from ovnkube-node-xzqcm...
2026-07-07T14:36:47.526429107Z Wrote inspect data to must-gather.
2026-07-07T14:36:47.657263685Z tar: Removing leading `/' from member names
2026-07-07T14:36:47.660438360Z tar: Removing leading `/' from member names
2026-07-07T14:36:47.670419437Z tar: Removing leading `/' from member names
2026-07-07T14:36:47.688743051Z tar: Removing leading `/' from member names
2026-07-07T14:36:47.708807095Z tar: Removing leading `/' from member names
2026-07-07T14:36:47.719447961Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T14:36:47.720656856Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T14:36:47.732090157Z tar: Removing leading `/' from member names
2026-07-07T14:36:47.839194635Z Wrote inspect data to must-gather.
2026-07-07T14:36:47.946841592Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:48.097916010Z Wrote inspect data to must-gather.
2026-07-07T14:36:48.160049572Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T14:36:48.194112788Z INFO: Waiting for network log collection to complete ...
2026-07-07T14:36:48.194169875Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T14:36:48.194693402Z INFO: Copying ovnk databases complete.
2026-07-07T14:36:48.194952228Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:48.196263756Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-07T14:36:48.197175947Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T14:36:48.199068277Z ovnk_database_store/
2026-07-07T14:36:48.199068277Z ovnk_database_store/ovnkube-node-ddpch_nbdb
2026-07-07T14:36:48.210140655Z ovnk_database_store/ovnkube-node-5jd26_sbdb
2026-07-07T14:36:48.234017797Z ovnk_database_store/ovnkube-node-5jd26_nbdb
2026-07-07T14:36:48.243240394Z ovnk_database_store/ovnkube-node-xzqcm_sbdb
2026-07-07T14:36:48.265657225Z ovnk_database_store/ovnkube-node-ddpch_sbdb
2026-07-07T14:36:48.287214645Z ovnk_database_store/ovnkube-node-xzqcm_nbdb
2026-07-07T14:36:48.298661876Z INFO: Network log collection complete.
2026-07-07T14:36:48.339779555Z Wrote inspect data to must-gather.
2026-07-07T14:36:48.413377583Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:48.473729941Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T14:36:48.554295226Z Wrote inspect data to must-gather.
2026-07-07T14:36:48.640248560Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:48.642586695Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T14:36:48.769687753Z Wrote inspect data to must-gather.
2026-07-07T14:36:48.808177993Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T14:36:48.838622599Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:49.065311665Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:49.069234942Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T14:36:49.082696655Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T14:36:49.084910051Z INFO: Fallback to identify the container image from release info
2026-07-07T14:36:49.297929743Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:49.352453386Z INFO: Getting status from alertmanager-main-0
2026-07-07T14:36:49.527263867Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:49.529891578Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-07T14:36:49.641341314Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T14:36:49.688382482Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:49.904075775Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:49.906958886Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-07T14:36:50.066216241Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T14:36:50.105929351Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:50.212972775Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T14:36:50.284818515Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:36:50.324871056Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:50.327851701Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-07T14:36:50.366723190Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T14:36:50.408441833Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:36:50.487427609Z Gathering data for ns/openshift-network-console...
2026-07-07T14:36:50.515279386Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:50.736994369Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:50.740120467Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-07T14:36:50.931973525Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:51.023134785Z Gathering data for ns/openshift-console-operator...
2026-07-07T14:36:51.142060460Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:51.144672236Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-07T14:36:51.344790854Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:51.522042688Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T14:36:51.523976846Z Gathering data for ns/openshift-console...
2026-07-07T14:36:51.565948019Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:51.568458432Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T14:36:51.742764053Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:51.930572944Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:36:52.200799884Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T14:36:52.660140200Z Done executing Istio gather script
2026-07-07T14:36:52.667858778Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T14:36:52.895591460Z Gathering data for ns/openshift-dns-operator...
2026-07-07T14:36:53.261202727Z Gathering data for ns/openshift-dns...
2026-07-07T14:36:53.793238422Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T14:36:54.108952034Z Gathering data for ns/openshift-image-registry...
2026-07-07T14:36:54.932473459Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T14:36:54.934496437Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T14:36:55.387123467Z Gathering data for ns/openshift-ingress...
2026-07-07T14:36:55.420535127Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:36:55.847155017Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T14:36:56.070850317Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T14:36:56.415019624Z Gathering data for ns/openshift-insights...
2026-07-07T14:36:57.192915352Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T14:36:58.311042561Z Waiting for performance profile collector pods to become ready: 8
2026-07-07T14:36:59.185353243Z Gathering data for ns/kserve...
2026-07-07T14:36:59.438111558Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T14:36:59.691310479Z Collecting performance related data for node ip-10-0-138-42.ec2.internal
2026-07-07T14:36:59.698131758Z Collecting performance related data for node ip-10-0-130-166.ec2.internal
2026-07-07T14:36:59.705238505Z Collecting performance related data for node ip-10-0-135-29.ec2.internal
2026-07-07T14:37:00.432634976Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:37:01.532384612Z Gathering data for ns/openshift-config...
2026-07-07T14:37:01.888279344Z Gathering data for ns/openshift-config-managed...
2026-07-07T14:37:02.516676417Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T14:37:03.103781459Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T14:37:03.482744937Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T14:37:03.898664192Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T14:37:04.294611681Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T14:37:04.650761433Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T14:37:05.111640255Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T14:37:05.438112563Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:37:05.674898818Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T14:37:06.075383826Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T14:37:06.706184244Z Gathering data for ns/openshift-multus...
2026-07-07T14:37:08.444715908Z Collecting kubelet logs for node ip-10-0-138-42.ec2.internal
2026-07-07T14:37:08.446790072Z Collecting kubelet logs for node ip-10-0-130-166.ec2.internal
2026-07-07T14:37:08.449554091Z Collecting kubelet logs for node ip-10-0-135-29.ec2.internal
2026-07-07T14:37:08.832493856Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T14:37:08.833535746Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-z7x97 namespace
2026-07-07T14:37:08.843854892Z INFO: Node performance data collection complete.
2026-07-07T14:37:10.442730707Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:37:10.653040461Z Gathering data for ns/openshift-host-network...
2026-07-07T14:37:11.070751272Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T14:37:11.765867673Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T14:37:12.272773218Z Gathering data for ns/openshift-network-operator...
2026-07-07T14:37:12.753214549Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T14:37:13.119838163Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T14:37:13.695982313Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T14:37:14.058957873Z Gathering data for ns/openshift-apiserver...
2026-07-07T14:37:14.591162010Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T14:37:15.012699549Z Gathering data for ns/openshift-controller-manager...
2026-07-07T14:37:15.406193626Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T14:37:15.447102445Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-07-07T14:37:16.301471587Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T14:37:16.693305826Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T14:37:17.109416697Z Gathering data for ns/openshift-service-ca...
2026-07-07T14:37:17.596797336Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T14:37:18.740575808Z Wrote inspect data to must-gather.
2026-07-07T14:37:18.740617844Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T14:37:18.740617844Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T14:37:18.878440177Z error: the server doesn't have a resource type "clusters"
2026-07-07T14:37:18.934095412Z Caches written to disk