2026-09-16T19:29:42.764626144Z [disk usage checker] Started
2026-09-16T19:29:42.769198175Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-16T19:29:42.973118845Z Gathering data for ns/openshift-cluster-version...
2026-09-16T19:29:43.126381653Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T19:29:43.233587570Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T19:29:43.336270119Z Gathering data for ns/default...
2026-09-16T19:29:43.739180397Z Gathering data for ns/openshift...
2026-09-16T19:29:44.215113164Z Gathering data for ns/kube-system...
2026-09-16T19:29:44.976319198Z Gathering data for ns/openshift-etcd...
2026-09-16T19:29:45.423401985Z Wrote inspect data to must-gather.
2026-09-16T19:29:45.423492554Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T19:29:45.423492554Z     namespaces "assisted-installer" not found
2026-09-16T19:29:45.594664468Z Waiting on subprocesses to finish execution.
2026-09-16T19:29:45.602302101Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-16T19:29:45.606102170Z Executing Istio gather script
2026-09-16T19:29:45.608210283Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-16T19:29:45.610791100Z INFO: Gathering HAProxy config files
2026-09-16T19:29:45.612883683Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-16T19:29:45.631659117Z INFO: Collecting host service logs for crio
2026-09-16T19:29:45.632189156Z INFO: Collecting host service logs for kubelet
2026-09-16T19:29:45.632503748Z INFO: Collecting host service logs for rpm-ostreed
2026-09-16T19:29:45.632795775Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-16T19:29:45.633171668Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-16T19:29:45.633529679Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-16T19:29:45.633933601Z INFO: Collecting host service logs for NetworkManager
2026-09-16T19:29:45.634366201Z INFO: Collecting host service logs for openvswitch
2026-09-16T19:29:45.634742973Z INFO: Collecting host service logs for ovs-configuration
2026-09-16T19:29:45.635082522Z INFO: Collecting host service logs for ovsdb-server
2026-09-16T19:29:45.635416981Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-16T19:29:45.635791534Z INFO: Waiting for worker host service log collection to complete ...
2026-09-16T19:29:45.706756223Z INFO: Waiting for node performance related collection to complete ...
2026-09-16T19:29:46.263484211Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-16T19:29:46.301419421Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-16T19:29:46.301508529Z INFO: on-disk MachineConfig config collection complete.
2026-09-16T19:29:46.641592870Z No resources found in openshift-etcd namespace.
2026-09-16T19:29:46.643032340Z INFO: "metallb-operator" not detected. Skipping.
2026-09-16T19:29:46.678444989Z No resources found
2026-09-16T19:29:46.792087574Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-16T19:29:46.867465678Z error: the server doesn't have a resource type "performanceprofile"
2026-09-16T19:29:46.923069077Z Wrote inspect data to must-gather.
2026-09-16T19:29:47.016582421Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-16T19:29:47.093093943Z INFO: Worker host service log collection to complete.
2026-09-16T19:29:47.197471078Z INFO: Collecting Insights Archives from insights-operator-5844b856db-dz57d
2026-09-16T19:29:47.197529549Z insights-runtime-extractor-5j2xj
2026-09-16T19:29:47.197548869Z insights-runtime-extractor-f2h8c
2026-09-16T19:29:47.197567729Z insights-runtime-extractor-pxv54
2026-09-16T19:29:47.215803976Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-16T19:29:47.602761199Z No resources found
2026-09-16T19:29:47.628746438Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-16T19:29:47.769938499Z Wrote inspect data to must-gather.
2026-09-16T19:29:47.779934012Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T19:29:47.780941076Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-16T19:29:47.822270711Z Wrote inspect data to must-gather.
2026-09-16T19:29:47.915868328Z ERROR: No running kube-apiserver pods found
2026-09-16T19:29:48.115471418Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-16T19:29:48.115471418Z See 'oc rsync -h' for help and examples
2026-09-16T19:29:48.177894625Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-16T19:29:48.194735463Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-16T19:29:48.235136363Z Wrote inspect data to must-gather.
2026-09-16T19:29:48.253234949Z Wrote inspect data to must-gather.
2026-09-16T19:29:48.394910531Z Wrote inspect data to must-gather.
2026-09-16T19:29:48.412764143Z error: the server doesn't have a resource type "machineconfigs"
2026-09-16T19:29:48.431450420Z Wrote inspect data to must-gather.
2026-09-16T19:29:48.445362221Z INFO: 'previous-logs' folder not found on ip-10-0-128-29.ec2.internal, skipping...
2026-09-16T19:29:48.491105853Z Gathering data for ns/openshift-monitoring...
2026-09-16T19:29:48.549437624Z tar: Removing leading `/' from member names
2026-09-16T19:29:48.559352432Z INFO: HAProxy config collection complete.
2026-09-16T19:29:48.659578348Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-16T19:29:48.691393580Z Wrote inspect data to must-gather.
2026-09-16T19:29:48.697570663Z INFO: Getting rules from prometheus-k8s-0
2026-09-16T19:29:48.780331809Z INFO: 'previous-logs' folder not found on ip-10-0-133-99.ec2.internal, skipping...
2026-09-16T19:29:48.912190763Z Wrote inspect data to must-gather.
2026-09-16T19:29:49.014111088Z Wrote inspect data to must-gather.
2026-09-16T19:29:49.108319573Z INFO: 'previous-logs' folder not found on ip-10-0-140-67.ec2.internal, skipping...
2026-09-16T19:29:49.108420183Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-16T19:29:49.108481158Z INFO: Machine Config Daemon termination log collection complete.
2026-09-16T19:29:49.117693379Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-16T19:29:49.195177345Z INFO: Getting status/config from prometheus-k8s-0
2026-09-16T19:29:49.317067012Z Wrote inspect data to must-gather.
2026-09-16T19:29:49.368024539Z Wrote inspect data to must-gather.
2026-09-16T19:29:49.614038523Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-16T19:29:49.635538531Z Wrote inspect data to must-gather.
2026-09-16T19:29:49.659592254Z INFO: INTERCONNECT MODE
2026-09-16T19:29:49.659619554Z INFO: Gathering ovn-kubernetes DBs
2026-09-16T19:29:49.680488147Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-16T19:29:49.834465131Z INFO: Gathering OVN_Northbound from ovnkube-node-b55tw...
2026-09-16T19:29:49.834743891Z INFO: Gathering OVN_Northbound from ovnkube-node-smblz...
2026-09-16T19:29:49.834967929Z INFO: Gathering OVN_Northbound from ovnkube-node-z7clt...
2026-09-16T19:29:49.835323596Z INFO: Gathering OVN_Southbound from ovnkube-node-b55tw...
2026-09-16T19:29:49.835636903Z INFO: Gathering OVN_Southbound from ovnkube-node-smblz...
2026-09-16T19:29:49.835862543Z INFO: Gathering OVN_Southbound from ovnkube-node-z7clt...
2026-09-16T19:29:49.879569849Z Wrote inspect data to must-gather.
2026-09-16T19:29:50.001417279Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-16T19:29:50.146901717Z tar: Removing leading `/' from member names
2026-09-16T19:29:50.151539432Z tar: Removing leading `/' from member names
2026-09-16T19:29:50.162323877Z tar: Removing leading `/' from member names
2026-09-16T19:29:50.183560670Z tar: Removing leading `/' from member names
2026-09-16T19:29:50.188320428Z tar: Removing leading `/' from member names
2026-09-16T19:29:50.206238828Z tar: Removing leading `/' from member names
2026-09-16T19:29:50.238655262Z Wrote inspect data to must-gather.
2026-09-16T19:29:50.341944226Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-16T19:29:50.518668583Z Wrote inspect data to must-gather.
2026-09-16T19:29:50.535269413Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T19:29:50.627872181Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-16T19:29:50.667764190Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-16T19:29:50.765283893Z Wrote inspect data to must-gather.
2026-09-16T19:29:50.768959980Z INFO: Waiting for network log collection to complete ...
2026-09-16T19:29:50.769007636Z INFO: Waiting for ovnk database copies to complete ...
2026-09-16T19:29:50.769502200Z INFO: Copying ovnk databases complete.
2026-09-16T19:29:50.771272597Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-09-16T19:29:50.774176828Z ovnk_database_store/
2026-09-16T19:29:50.774176828Z ovnk_database_store/ovnkube-node-smblz_nbdb
2026-09-16T19:29:50.788977937Z ovnk_database_store/ovnkube-node-z7clt_nbdb
2026-09-16T19:29:50.802418202Z ovnk_database_store/ovnkube-node-b55tw_nbdb
2026-09-16T19:29:50.811787671Z ovnk_database_store/ovnkube-node-smblz_sbdb
2026-09-16T19:29:50.843200473Z ovnk_database_store/ovnkube-node-z7clt_sbdb
2026-09-16T19:29:50.850221591Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:50.866382239Z ovnk_database_store/ovnkube-node-b55tw_sbdb
2026-09-16T19:29:50.893301804Z INFO: Network log collection complete.
2026-09-16T19:29:51.004537084Z Wrote inspect data to must-gather.
2026-09-16T19:29:51.096992278Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:51.100337399Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-16T19:29:51.143144042Z INFO: Getting status from alertmanager-main-0
2026-09-16T19:29:51.235995876Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T19:29:51.314399516Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-16T19:29:51.317011212Z INFO: Fallback to identify the container image from release info
2026-09-16T19:29:51.322568880Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:51.469825350Z Gathering data for ns/openshift-network-console...
2026-09-16T19:29:51.532205712Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:51.535259790Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-16T19:29:51.725433483Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:51.847003352Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-16T19:29:51.957364528Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:51.959731989Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-16T19:29:52.007104855Z daemonset.apps/perf-node-gather-daemonset created
2026-09-16T19:29:52.068053526Z Gathering data for ns/openshift-console-operator...
2026-09-16T19:29:52.165691144Z Waiting for performance profile collector pods to become ready: 1
2026-09-16T19:29:52.218897973Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:52.401926479Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:52.405487473Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-09-16T19:29:52.601672943Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:52.604981698Z Gathering data for ns/openshift-console...
2026-09-16T19:29:52.788652919Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-16T19:29:52.799374173Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:52.802108649Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-09-16T19:29:52.948444288Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:53.141497015Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:53.143808599Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-16T19:29:53.301920391Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-16T19:29:53.308556956Z Waiting for performance profile collector pods to become ready: 2
2026-09-16T19:29:53.335436198Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:53.535693449Z Wrote inspect data to /must-gather/istio.
2026-09-16T19:29:53.943059730Z Gathering data for ns/openshift-dns-operator...
2026-09-16T19:29:54.275570541Z Done executing Istio gather script
2026-09-16T19:29:54.344923001Z Gathering data for ns/openshift-dns...
2026-09-16T19:29:54.432587153Z Waiting for performance profile collector pods to become ready: 3
2026-09-16T19:29:55.039586201Z Gathering data for ns/openshift-image-registry...
2026-09-16T19:29:55.562167911Z Waiting for performance profile collector pods to become ready: 4
2026-09-16T19:29:55.614819094Z Gathering data for ns/openshift-ingress-operator...
2026-09-16T19:29:56.026426168Z Gathering data for ns/openshift-ingress...
2026-09-16T19:29:56.403841566Z Gathering data for ns/openshift-ingress-canary...
2026-09-16T19:29:56.676996480Z Waiting for performance profile collector pods to become ready: 5
2026-09-16T19:29:56.861873311Z Gathering data for ns/openshift-insights...
2026-09-16T19:29:57.793932794Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-16T19:29:57.795384474Z Waiting for performance profile collector pods to become ready: 6
2026-09-16T19:29:58.910276473Z Waiting for performance profile collector pods to become ready: 7
2026-09-16T19:29:59.521515808Z Gathering data for ns/kserve...
2026-09-16T19:30:00.020504583Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-16T19:30:00.301133282Z Collecting performance related data for node ip-10-0-128-29.ec2.internal
2026-09-16T19:30:00.308332413Z Collecting performance related data for node ip-10-0-140-67.ec2.internal
2026-09-16T19:30:00.316085082Z Collecting performance related data for node ip-10-0-133-99.ec2.internal
2026-09-16T19:30:01.470756795Z Gathering data for ns/openshift-config...
2026-09-16T19:30:01.893714218Z Gathering data for ns/openshift-config-managed...
2026-09-16T19:30:02.390271976Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-16T19:30:02.765687626Z Gathering data for ns/openshift-kube-apiserver...
2026-09-16T19:30:02.803560374Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-16T19:30:03.149095561Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-16T19:30:03.505725937Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-16T19:30:03.895042091Z Gathering data for ns/openshift-kube-scheduler...
2026-09-16T19:30:04.277709338Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-16T19:30:04.655083466Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-16T19:30:05.082066728Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-16T19:30:05.460907960Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-16T19:30:06.089845279Z Gathering data for ns/openshift-multus...
2026-09-16T19:30:07.809381653Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-16T19:30:07.896787762Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-16T19:30:08.462245730Z Collecting kubelet logs for node ip-10-0-128-29.ec2.internal
2026-09-16T19:30:08.464199002Z Collecting kubelet logs for node ip-10-0-140-67.ec2.internal
2026-09-16T19:30:08.466026225Z Collecting kubelet logs for node ip-10-0-133-99.ec2.internal
2026-09-16T19:30:08.827525182Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-sc8ck namespace
2026-09-16T19:30:08.834823917Z INFO: Node performance data collection complete.
2026-09-16T19:30:09.391258477Z Gathering data for ns/openshift-host-network...
2026-09-16T19:30:09.768840454Z Gathering data for ns/openshift-network-diagnostics...
2026-09-16T19:30:10.383883010Z Gathering data for ns/openshift-network-node-identity...
2026-09-16T19:30:10.865546917Z Gathering data for ns/openshift-network-operator...
2026-09-16T19:30:11.336873491Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-16T19:30:11.643696334Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-16T19:30:12.251517796Z Gathering data for ns/openshift-apiserver-operator...
2026-09-16T19:30:12.607753665Z Gathering data for ns/openshift-apiserver...
2026-09-16T19:30:12.814354522Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-09-16T19:30:13.097251843Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-16T19:30:13.453847447Z Gathering data for ns/openshift-controller-manager...
2026-09-16T19:30:13.812667808Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-16T19:30:14.545371074Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-16T19:30:15.015136893Z Gathering data for ns/openshift-service-ca-operator...
2026-09-16T19:30:15.451131451Z Gathering data for ns/openshift-service-ca...
2026-09-16T19:30:15.881669809Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-16T19:30:16.994337960Z Wrote inspect data to must-gather.
2026-09-16T19:30:16.994392822Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T19:30:16.994392822Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-16T19:30:17.132851326Z error: the server doesn't have a resource type "clusters"
2026-09-16T19:30:17.281035480Z Caches written to disk