2026-06-03T16:00:33.894787555Z [disk usage checker] Started
2026-06-03T16:00:33.898557960Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:00:34.131987470Z Gathering data for ns/openshift-cluster-version...
2026-06-03T16:00:34.310829573Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-03T16:00:34.426329377Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-03T16:00:34.534771009Z Gathering data for ns/default...
2026-06-03T16:00:35.055394825Z Gathering data for ns/openshift...
2026-06-03T16:00:35.562089595Z Gathering data for ns/kube-system...
2026-06-03T16:00:36.357597089Z Gathering data for ns/openshift-etcd...
2026-06-03T16:00:36.366869337Z Waiting on subprocesses to finish execution.
2026-06-03T16:00:36.376015637Z INFO: Gathering HAProxy config files
2026-06-03T16:00:36.387226840Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-03T16:00:36.392278389Z INFO: Collecting host service logs for crio
2026-06-03T16:00:36.392689352Z INFO: Collecting host service logs for kubelet
2026-06-03T16:00:36.393043612Z INFO: Collecting host service logs for rpm-ostreed
2026-06-03T16:00:36.393555018Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-03T16:00:36.393900656Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-03T16:00:36.394239267Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-03T16:00:36.394559826Z INFO: Collecting host service logs for NetworkManager
2026-06-03T16:00:36.394870002Z INFO: Collecting host service logs for openvswitch
2026-06-03T16:00:36.395189748Z INFO: Collecting host service logs for ovs-configuration
2026-06-03T16:00:36.395546037Z INFO: Collecting host service logs for ovsdb-server
2026-06-03T16:00:36.400637554Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-03T16:00:36.401042828Z INFO: Waiting for worker host service log collection to complete ...
2026-06-03T16:00:36.405849592Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-03T16:00:36.406328125Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-03T16:00:36.563831653Z INFO: Waiting for node performance related collection to complete ...
2026-06-03T16:00:37.059419671Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-03T16:00:37.135479661Z INFO: "metallb-operator" not detected. Skipping.
2026-06-03T16:00:37.149460209Z INFO: Collecting Insights Archives from insights-operator-788bf58df5-54mdl
2026-06-03T16:00:37.149502790Z insights-runtime-extractor-k2lgj
2026-06-03T16:00:37.149535056Z insights-runtime-extractor-l22gp
2026-06-03T16:00:37.149557161Z insights-runtime-extractor-scnr2
2026-06-03T16:00:37.513104831Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-03T16:00:37.762830861Z error: the server doesn't have a resource type "performanceprofile"
2026-06-03T16:00:37.782560210Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-03T16:00:37.782560210Z See 'oc rsync -h' for help and examples
2026-06-03T16:00:37.791145959Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-03T16:00:37.972719245Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-03T16:00:38.016143389Z INFO: Worker host service log collection to complete.
2026-06-03T16:00:38.078812785Z No resources found in openshift-etcd namespace.
2026-06-03T16:00:38.153690961Z No resources found
2026-06-03T16:00:38.178700602Z No resources found
2026-06-03T16:00:38.345912910Z ERROR: No running kube-apiserver pods found
2026-06-03T16:00:38.737551076Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-03T16:00:38.781691265Z Wrote inspect data to must-gather.
2026-06-03T16:00:38.891610613Z Wrote inspect data to must-gather.
2026-06-03T16:00:38.891650911Z error: inspection completed with the errors occurred while gathering data:
2026-06-03T16:00:38.891650911Z     namespaces "assisted-installer" not found
2026-06-03T16:00:38.898294763Z Wrote inspect data to must-gather.
2026-06-03T16:00:38.908669545Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-03T16:00:38.908745728Z INFO: on-disk MachineConfig config collection complete.
2026-06-03T16:00:38.948713432Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:00:38.997606056Z Wrote inspect data to must-gather.
2026-06-03T16:00:39.091383701Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-03T16:00:39.091383701Z Wrote inspect data to must-gather.
2026-06-03T16:00:39.224534410Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-03T16:00:39.227609947Z Wrote inspect data to must-gather.
2026-06-03T16:00:39.302756742Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-03T16:00:39.328667693Z Wrote inspect data to must-gather.
2026-06-03T16:00:39.360165226Z INFO: 'previous-logs' folder not found on ip-10-0-128-175.ec2.internal, skipping...
2026-06-03T16:00:39.531830093Z Wrote inspect data to must-gather.
2026-06-03T16:00:39.546591197Z error: the server doesn't have a resource type "machineconfigs"
2026-06-03T16:00:39.561074399Z tar: Removing leading `/' from member names
2026-06-03T16:00:39.567612000Z INFO: HAProxy config collection complete.
2026-06-03T16:00:39.763472812Z INFO: 'previous-logs' folder not found on ip-10-0-129-129.ec2.internal, skipping...
2026-06-03T16:00:39.771776693Z INFO: Getting rules from prometheus-k8s-0
2026-06-03T16:00:39.774974826Z Gathering data for ns/kuadrant-system...
2026-06-03T16:00:39.964742609Z Wrote inspect data to must-gather.
2026-06-03T16:00:39.978343544Z Wrote inspect data to must-gather.
2026-06-03T16:00:40.052034814Z INFO: 'previous-logs' folder not found on ip-10-0-139-233.ec2.internal, skipping...
2026-06-03T16:00:40.052074279Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-03T16:00:40.052102084Z INFO: Machine Config Daemon termination log collection complete.
2026-06-03T16:00:40.273330199Z INFO: Getting status/config from prometheus-k8s-0
2026-06-03T16:00:40.332497617Z Wrote inspect data to must-gather.
2026-06-03T16:00:40.430855678Z Wrote inspect data to must-gather.
2026-06-03T16:00:40.533737067Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-03T16:00:40.789440466Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-03T16:00:40.823645755Z INFO: INTERCONNECT MODE
2026-06-03T16:00:40.823692490Z INFO: Gathering ovn-kubernetes DBs
2026-06-03T16:00:40.852257133Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-03T16:00:40.936253453Z Wrote inspect data to must-gather.
2026-06-03T16:00:41.036906222Z INFO: Gathering OVN_Northbound from ovnkube-node-2mdjq...
2026-06-03T16:00:41.037327190Z INFO: Gathering OVN_Northbound from ovnkube-node-4gqgl...
2026-06-03T16:00:41.037387656Z INFO: Gathering OVN_Northbound from ovnkube-node-kdw5d...
2026-06-03T16:00:41.037667996Z INFO: Gathering OVN_Southbound from ovnkube-node-2mdjq...
2026-06-03T16:00:41.038104228Z INFO: Gathering OVN_Southbound from ovnkube-node-4gqgl...
2026-06-03T16:00:41.038238954Z INFO: Gathering OVN_Southbound from ovnkube-node-kdw5d...
2026-06-03T16:00:41.093526899Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-03T16:00:41.367401785Z tar: Removing leading `/' from member names
2026-06-03T16:00:41.419384911Z tar: Removing leading `/' from member names
2026-06-03T16:00:41.425663616Z tar: Removing leading `/' from member names
2026-06-03T16:00:41.480523139Z W0603 16:00:41.480470    1195 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-03T16:00:41.505244865Z tar: Removing leading `/' from member names
2026-06-03T16:00:41.537922955Z Wrote inspect data to must-gather.
2026-06-03T16:00:41.558651588Z tar: Removing leading `/' from member names
2026-06-03T16:00:41.569984043Z tar: Removing leading `/' from member names
2026-06-03T16:00:41.607169596Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-03T16:00:41.638973002Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-03T16:00:41.891426804Z Gathering data for ns/openshift-monitoring...
2026-06-03T16:00:41.969948420Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-03T16:00:42.016498252Z Wrote inspect data to must-gather.
2026-06-03T16:00:42.215326846Z INFO: Waiting for network log collection to complete ...
2026-06-03T16:00:42.215748344Z INFO: Waiting for ovnk database copies to complete ...
2026-06-03T16:00:42.216434074Z INFO: Copying ovnk databases complete.
2026-06-03T16:00:42.218343027Z 4.3M	must-gather/network_logs/ovnk_database_store
2026-06-03T16:00:42.221522848Z ovnk_database_store/
2026-06-03T16:00:42.221579867Z ovnk_database_store/ovnkube-node-kdw5d_sbdb
2026-06-03T16:00:42.249612333Z ovnk_database_store/ovnkube-node-4gqgl_sbdb
2026-06-03T16:00:42.281419028Z ovnk_database_store/ovnkube-node-kdw5d_nbdb
2026-06-03T16:00:42.291231572Z ovnk_database_store/ovnkube-node-2mdjq_sbdb
2026-06-03T16:00:42.314940462Z ovnk_database_store/ovnkube-node-4gqgl_nbdb
2026-06-03T16:00:42.326633559Z ovnk_database_store/ovnkube-node-2mdjq_nbdb
2026-06-03T16:00:42.338759418Z INFO: Network log collection complete.
2026-06-03T16:00:42.461849277Z Wrote inspect data to must-gather.
2026-06-03T16:00:42.488283767Z INFO: Getting status from alertmanager-main-0
2026-06-03T16:00:42.837459537Z Wrote inspect data to must-gather.
2026-06-03T16:00:43.174845905Z Wrote inspect data to must-gather.
2026-06-03T16:00:43.531851549Z Wrote inspect data to must-gather.
2026-06-03T16:00:43.807413432Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-03T16:00:43.809774569Z INFO: Fallback to identify the container image from release info
2026-06-03T16:00:43.973752136Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:00:44.311708937Z Gathering data for ns/openshift-network-console...
2026-06-03T16:00:44.537159660Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8381cb85e479760bd6b601e111b39e6c56a632bfa298867f4197ec4421e9f5
2026-06-03T16:00:44.688167239Z daemonset.apps/perf-node-gather-daemonset created
2026-06-03T16:00:44.857834138Z Waiting for performance profile collector pods to become ready: 1
2026-06-03T16:00:44.891932339Z Gathering data for ns/openshift-console-operator...
2026-06-03T16:00:45.403528865Z Gathering data for ns/openshift-console...
2026-06-03T16:00:46.009534195Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-03T16:00:46.021925617Z Waiting for performance profile collector pods to become ready: 2
2026-06-03T16:00:46.402979217Z Wrote inspect data to must-gather.
2026-06-03T16:00:46.459330130Z Gathering data for ns/openshift-dns-operator...
2026-06-03T16:00:46.829029203Z Gathering data for ns/openshift-dns...
2026-06-03T16:00:47.149455720Z Waiting for performance profile collector pods to become ready: 3
2026-06-03T16:00:47.627848045Z Gathering data for ns/openshift-image-registry...
2026-06-03T16:00:48.167819827Z Gathering data for ns/openshift-ingress-operator...
2026-06-03T16:00:48.291527915Z Waiting for performance profile collector pods to become ready: 4
2026-06-03T16:00:48.601737655Z Gathering data for ns/openshift-ingress...
2026-06-03T16:00:48.981712124Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:00:49.430975845Z Waiting for performance profile collector pods to become ready: 5
2026-06-03T16:00:49.442007904Z Gathering data for ns/openshift-ingress-canary...
2026-06-03T16:00:49.942506156Z Gathering data for ns/openshift-insights...
2026-06-03T16:00:50.563407443Z Waiting for performance profile collector pods to become ready: 6
2026-06-03T16:00:51.685486769Z Waiting for performance profile collector pods to become ready: 7
2026-06-03T16:00:52.305026606Z Gathering data for ns/opendatahub...
2026-06-03T16:00:52.825512258Z Waiting for performance profile collector pods to become ready: 8
2026-06-03T16:00:53.958462008Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-03T16:00:53.986409227Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:00:54.208826196Z Collecting performance related data for node ip-10-0-139-233.ec2.internal
2026-06-03T16:00:54.215755107Z Collecting performance related data for node ip-10-0-129-129.ec2.internal
2026-06-03T16:00:54.223221181Z Collecting performance related data for node ip-10-0-128-175.ec2.internal
2026-06-03T16:00:54.285038965Z Gathering data for ns/openshift-lws-operator...
2026-06-03T16:00:56.178793821Z Gathering data for ns/openshift-config...
2026-06-03T16:00:56.555495652Z Gathering data for ns/openshift-config-managed...
2026-06-03T16:00:57.132050579Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-03T16:00:57.532884917Z Gathering data for ns/openshift-kube-apiserver...
2026-06-03T16:00:57.877444159Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-03T16:00:58.268400172Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-03T16:00:58.653467320Z Gathering data for ns/openshift-kube-scheduler...
2026-06-03T16:00:58.991007008Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:00:59.110660898Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-03T16:00:59.514056571Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-03T16:00:59.924877561Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-03T16:01:00.322643111Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-03T16:01:01.160419391Z Gathering data for ns/openshift-multus...
2026-06-03T16:01:02.774678423Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-03T16:01:02.840149391Z Collecting kubelet logs for node ip-10-0-139-233.ec2.internal
2026-06-03T16:01:02.841936254Z Collecting kubelet logs for node ip-10-0-129-129.ec2.internal
2026-06-03T16:01:02.844055000Z Collecting kubelet logs for node ip-10-0-128-175.ec2.internal
2026-06-03T16:01:03.175162953Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-06-03T16:01:03.180953356Z INFO: Node performance data collection complete.
2026-06-03T16:01:03.996565835Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:01:04.552643833Z Gathering data for ns/openshift-host-network...
2026-06-03T16:01:04.974697288Z Gathering data for ns/openshift-network-diagnostics...
2026-06-03T16:01:05.657646676Z Gathering data for ns/openshift-network-node-identity...
2026-06-03T16:01:06.240390820Z Gathering data for ns/openshift-network-operator...
2026-06-03T16:01:06.711173744Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-03T16:01:07.054366383Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-03T16:01:07.598737719Z Gathering data for ns/openshift-apiserver-operator...
2026-06-03T16:01:07.980452193Z Gathering data for ns/openshift-apiserver...
2026-06-03T16:01:08.463218143Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-03T16:01:08.780566167Z Gathering data for ns/openshift-controller-manager...
2026-06-03T16:01:09.008338641Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-06-03T16:01:09.171737198Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-03T16:01:09.926969148Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-03T16:01:10.296159943Z Gathering data for ns/openshift-service-ca-operator...
2026-06-03T16:01:10.672156728Z Gathering data for ns/openshift-service-ca...
2026-06-03T16:01:11.112560145Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-03T16:01:12.234551898Z Wrote inspect data to must-gather.
2026-06-03T16:01:12.234595506Z error: inspection completed with the errors occurred while gathering data:
2026-06-03T16:01:12.234595506Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-03T16:01:12.370462279Z error: the server doesn't have a resource type "clusters"
2026-06-03T16:01:12.553596644Z Caches written to disk