2026-05-28T16:42:21.239582059Z [disk usage checker] Started
2026-05-28T16:42:21.243624268Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T16:42:21.469767801Z Gathering data for ns/openshift-cluster-version...
2026-05-28T16:42:21.627952486Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T16:42:21.715539233Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T16:42:21.843990139Z Gathering data for ns/default...
2026-05-28T16:42:22.206394227Z Gathering data for ns/openshift...
2026-05-28T16:42:22.658469356Z Gathering data for ns/kube-system...
2026-05-28T16:42:23.382336999Z Gathering data for ns/openshift-etcd...
2026-05-28T16:42:23.572478763Z Waiting on subprocesses to finish execution.
2026-05-28T16:42:23.586434628Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T16:42:23.588991322Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T16:42:23.594098527Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T16:42:23.595195241Z INFO: Gathering HAProxy config files
2026-05-28T16:42:23.605110707Z INFO: Collecting host service logs for crio
2026-05-28T16:42:23.605481142Z INFO: Collecting host service logs for kubelet
2026-05-28T16:42:23.605868213Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T16:42:23.606221591Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T16:42:23.606587439Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T16:42:23.606962466Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T16:42:23.607337620Z INFO: Collecting host service logs for NetworkManager
2026-05-28T16:42:23.607720652Z INFO: Collecting host service logs for openvswitch
2026-05-28T16:42:23.608091500Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T16:42:23.608464368Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T16:42:23.608834411Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T16:42:23.609254139Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T16:42:23.828027457Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T16:42:24.224057298Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T16:42:24.306985551Z Wrote inspect data to must-gather.
2026-05-28T16:42:24.307071421Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T16:42:24.307071421Z     namespaces "assisted-installer" not found
2026-05-28T16:42:24.433122990Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T16:42:24.441221615Z No resources found
2026-05-28T16:42:24.441643357Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T16:42:24.478294246Z No resources found in openshift-etcd namespace.
2026-05-28T16:42:24.832999439Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-gq7tm
2026-05-28T16:42:24.833089027Z insights-runtime-extractor-5t4th
2026-05-28T16:42:24.833110102Z insights-runtime-extractor-7x5pb
2026-05-28T16:42:24.833129444Z insights-runtime-extractor-m587s
2026-05-28T16:42:24.982088915Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T16:42:25.002596360Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T16:42:25.014165666Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T16:42:25.140254925Z No resources found
2026-05-28T16:42:25.298860922Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T16:42:25.360062050Z INFO: Worker host service log collection to complete.
2026-05-28T16:42:25.407772531Z Wrote inspect data to must-gather.
2026-05-28T16:42:25.415145697Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T16:42:25.415230134Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T16:42:25.566916767Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T16:42:25.697303720Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T16:42:25.795639171Z INFO: 'previous-logs' folder not found on ip-10-0-132-51.ec2.internal, skipping...
2026-05-28T16:42:25.854539796Z Wrote inspect data to must-gather.
2026-05-28T16:42:25.922330205Z Wrote inspect data to must-gather.
2026-05-28T16:42:26.054057098Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T16:42:26.153200001Z ERROR: No running kube-apiserver pods found
2026-05-28T16:42:26.172787272Z Wrote inspect data to must-gather.
2026-05-28T16:42:26.200843509Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T16:42:26.200843509Z See 'oc rsync -h' for help and examples
2026-05-28T16:42:26.287106344Z Wrote inspect data to must-gather.
2026-05-28T16:42:26.295328818Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T16:42:26.373554127Z Wrote inspect data to must-gather.
2026-05-28T16:42:26.463135347Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T16:42:26.509585168Z Wrote inspect data to must-gather.
2026-05-28T16:42:26.551045074Z tar: Removing leading `/' from member names
2026-05-28T16:42:26.558443221Z INFO: HAProxy config collection complete.
2026-05-28T16:42:26.580171885Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T16:42:26.592961487Z INFO: 'previous-logs' folder not found on ip-10-0-134-100.ec2.internal, skipping...
2026-05-28T16:42:26.785558092Z Wrote inspect data to must-gather.
2026-05-28T16:42:26.889981548Z Gathering data for ns/kuadrant-system...
2026-05-28T16:42:26.891193501Z Wrote inspect data to must-gather.
2026-05-28T16:42:26.919713278Z INFO: 'previous-logs' folder not found on ip-10-0-143-102.ec2.internal, skipping...
2026-05-28T16:42:26.919815096Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T16:42:26.919870642Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T16:42:27.081033282Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T16:42:27.087134915Z Wrote inspect data to must-gather.
2026-05-28T16:42:27.211050628Z Wrote inspect data to must-gather.
2026-05-28T16:42:27.308159867Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T16:42:27.519344848Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T16:42:27.583218844Z INFO: INTERCONNECT MODE
2026-05-28T16:42:27.583315834Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T16:42:27.600098215Z Wrote inspect data to must-gather.
2026-05-28T16:42:27.790060325Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T16:42:27.798322815Z INFO: Gathering OVN_Northbound from ovnkube-node-28pqt...
2026-05-28T16:42:27.798657473Z INFO: Gathering OVN_Northbound from ovnkube-node-db77q...
2026-05-28T16:42:27.798947178Z INFO: Gathering OVN_Northbound from ovnkube-node-z9v5g...
2026-05-28T16:42:27.799450263Z INFO: Gathering OVN_Southbound from ovnkube-node-28pqt...
2026-05-28T16:42:27.799724204Z INFO: Gathering OVN_Southbound from ovnkube-node-db77q...
2026-05-28T16:42:27.800009601Z INFO: Gathering OVN_Southbound from ovnkube-node-z9v5g...
2026-05-28T16:42:27.955477914Z Wrote inspect data to must-gather.
2026-05-28T16:42:28.012523453Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T16:42:28.126423142Z tar: Removing leading `/' from member names
2026-05-28T16:42:28.133412087Z tar: Removing leading `/' from member names
2026-05-28T16:42:28.165786358Z tar: Removing leading `/' from member names
2026-05-28T16:42:28.219715551Z tar: Removing leading `/' from member names
2026-05-28T16:42:28.289524986Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T16:42:28.312615275Z tar: Removing leading `/' from member names
2026-05-28T16:42:28.316869696Z tar: Removing leading `/' from member names
2026-05-28T16:42:28.441250358Z Wrote inspect data to must-gather.
2026-05-28T16:42:28.626562106Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T16:42:28.714850737Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T16:42:28.866768503Z Wrote inspect data to must-gather.
2026-05-28T16:42:28.948095815Z INFO: Waiting for network log collection to complete ...
2026-05-28T16:42:28.948126945Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T16:42:28.948634022Z INFO: Copying ovnk databases complete.
2026-05-28T16:42:28.950524014Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-05-28T16:42:28.953673627Z ovnk_database_store/
2026-05-28T16:42:28.953722385Z ovnk_database_store/ovnkube-node-28pqt_nbdb
2026-05-28T16:42:28.975717573Z Gathering data for ns/openshift-monitoring...
2026-05-28T16:42:29.003529320Z ovnk_database_store/ovnkube-node-db77q_sbdb
2026-05-28T16:42:29.031400743Z ovnk_database_store/ovnkube-node-28pqt_sbdb
2026-05-28T16:42:29.062228392Z ovnk_database_store/ovnkube-node-z9v5g_nbdb
2026-05-28T16:42:29.074197962Z ovnk_database_store/ovnkube-node-z9v5g_sbdb
2026-05-28T16:42:29.102070425Z ovnk_database_store/ovnkube-node-db77q_nbdb
2026-05-28T16:42:29.104784497Z INFO: Getting status from alertmanager-main-0
2026-05-28T16:42:29.117016241Z INFO: Network log collection complete.
2026-05-28T16:42:29.243297945Z Wrote inspect data to must-gather.
2026-05-28T16:42:29.562630781Z Wrote inspect data to must-gather.
2026-05-28T16:42:29.836398674Z Wrote inspect data to must-gather.
2026-05-28T16:42:30.148080465Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T16:42:30.150435286Z INFO: Fallback to identify the container image from release info
2026-05-28T16:42:30.673268334Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T16:42:30.812461591Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T16:42:30.969598353Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T16:42:30.980324713Z Gathering data for ns/openshift-network-console...
2026-05-28T16:42:31.314707180Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T16:42:31.486208876Z Gathering data for ns/openshift-console-operator...
2026-05-28T16:42:31.908811105Z Gathering data for ns/openshift-console...
2026-05-28T16:42:32.113377664Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T16:42:32.441045638Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T16:42:32.816009348Z Gathering data for ns/openshift-dns-operator...
2026-05-28T16:42:32.925166877Z Wrote inspect data to must-gather.
2026-05-28T16:42:33.133818521Z Gathering data for ns/openshift-dns...
2026-05-28T16:42:33.223745030Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T16:42:33.726200986Z Gathering data for ns/openshift-image-registry...
2026-05-28T16:42:34.179251059Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T16:42:34.337474205Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T16:42:34.533575307Z Gathering data for ns/openshift-ingress...
2026-05-28T16:42:35.290733714Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T16:42:35.470074642Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T16:42:35.727963708Z Gathering data for ns/openshift-insights...
2026-05-28T16:42:36.326958694Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T16:42:36.581181524Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T16:42:37.598878174Z Gathering data for ns/opendatahub...
2026-05-28T16:42:37.692869226Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T16:42:38.810761811Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T16:42:38.908731582Z Gathering data for ns/openshift-lws-operator...
2026-05-28T16:42:39.075959782Z Collecting performance related data for node ip-10-0-143-102.ec2.internal
2026-05-28T16:42:39.082934876Z Collecting performance related data for node ip-10-0-134-100.ec2.internal
2026-05-28T16:42:39.091145421Z Collecting performance related data for node ip-10-0-132-51.ec2.internal
2026-05-28T16:42:40.311959507Z Gathering data for ns/openshift-config...
2026-05-28T16:42:40.657294675Z Gathering data for ns/openshift-config-managed...
2026-05-28T16:42:41.091968639Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T16:42:41.333137315Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T16:42:41.394428074Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T16:42:41.714730386Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T16:42:42.058865618Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T16:42:42.381292607Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T16:42:42.664153750Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T16:42:42.972626856Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T16:42:43.305311238Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T16:42:43.637952714Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T16:42:44.180458778Z Gathering data for ns/openshift-multus...
2026-05-28T16:42:45.499395909Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T16:42:46.339742552Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T16:42:47.029001468Z Gathering data for ns/openshift-host-network...
2026-05-28T16:42:47.326198546Z Collecting kubelet logs for node ip-10-0-143-102.ec2.internal
2026-05-28T16:42:47.328645911Z Collecting kubelet logs for node ip-10-0-134-100.ec2.internal
2026-05-28T16:42:47.330789136Z Collecting kubelet logs for node ip-10-0-132-51.ec2.internal
2026-05-28T16:42:47.406437875Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T16:42:47.695070417Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T16:42:47.699699270Z INFO: Node performance data collection complete.
2026-05-28T16:42:48.032602842Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T16:42:48.447045762Z Gathering data for ns/openshift-network-operator...
2026-05-28T16:42:48.811377205Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T16:42:49.091827884Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T16:42:49.563612366Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T16:42:49.887319283Z Gathering data for ns/openshift-apiserver...
2026-05-28T16:42:50.321495357Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T16:42:50.598575940Z Gathering data for ns/openshift-controller-manager...
2026-05-28T16:42:50.909041740Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T16:42:51.348729908Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T16:42:51.534313324Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T16:42:51.821343364Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T16:42:52.134114038Z Gathering data for ns/openshift-service-ca...
2026-05-28T16:42:52.539957412Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T16:42:53.455249024Z Wrote inspect data to must-gather.
2026-05-28T16:42:53.455293903Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T16:42:53.455293903Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T16:42:53.583531965Z error: the server doesn't have a resource type "clusters"
2026-05-28T16:42:53.761946228Z Caches written to disk