2026-05-27T18:33:45.297058346Z [disk usage checker] Started
2026-05-27T18:33:45.300473493Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:33:45.536603633Z Gathering data for ns/openshift-cluster-version...
2026-05-27T18:33:45.716271765Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-27T18:33:45.827421829Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-27T18:33:45.937020976Z Gathering data for ns/default...
2026-05-27T18:33:46.292083864Z Gathering data for ns/openshift...
2026-05-27T18:33:46.761733377Z Gathering data for ns/kube-system...
2026-05-27T18:33:47.557245270Z Gathering data for ns/openshift-etcd...
2026-05-27T18:33:47.835413620Z Waiting on subprocesses to finish execution.
2026-05-27T18:33:47.844186977Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-27T18:33:47.846175349Z INFO: Collecting host service logs for crio
2026-05-27T18:33:47.846522254Z INFO: Collecting host service logs for kubelet
2026-05-27T18:33:47.846881320Z INFO: Collecting host service logs for rpm-ostreed
2026-05-27T18:33:47.847230458Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-27T18:33:47.847575893Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-27T18:33:47.847926274Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-27T18:33:47.848270195Z INFO: Collecting host service logs for NetworkManager
2026-05-27T18:33:47.848601991Z INFO: Collecting host service logs for openvswitch
2026-05-27T18:33:47.848946864Z INFO: Collecting host service logs for ovs-configuration
2026-05-27T18:33:47.849374303Z INFO: Collecting host service logs for ovsdb-server
2026-05-27T18:33:47.849754700Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-27T18:33:47.850164097Z INFO: Waiting for worker host service log collection to complete ...
2026-05-27T18:33:47.860817866Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-27T18:33:47.867029813Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-27T18:33:47.888815793Z INFO: Gathering HAProxy config files
2026-05-27T18:33:47.906338380Z INFO: Waiting for node performance related collection to complete ...
2026-05-27T18:33:48.280392737Z Wrote inspect data to must-gather.
2026-05-27T18:33:48.280441156Z error: inspection completed with the errors occurred while gathering data:
2026-05-27T18:33:48.280441156Z     namespaces "assisted-installer" not found
2026-05-27T18:33:48.386625398Z No resources found in openshift-etcd namespace.
2026-05-27T18:33:48.703958707Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-27T18:33:48.736666745Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-27T18:33:48.857949073Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-27T18:33:49.077563843Z INFO: "metallb-operator" not detected. Skipping.
2026-05-27T18:33:49.130421918Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-27T18:33:49.141281893Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-27T18:33:49.191753746Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-xqqn7
2026-05-27T18:33:49.191805108Z insights-runtime-extractor-664ml
2026-05-27T18:33:49.191827123Z insights-runtime-extractor-tr2v9
2026-05-27T18:33:49.191868056Z insights-runtime-extractor-xv57g
2026-05-27T18:33:49.369405695Z No resources found
2026-05-27T18:33:49.446985211Z error: the server doesn't have a resource type "performanceprofile"
2026-05-27T18:33:49.505198150Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-27T18:33:49.509102107Z INFO: on-disk MachineConfig config collection complete.
2026-05-27T18:33:49.694758297Z INFO: Worker host service log collection to complete.
2026-05-27T18:33:49.754576264Z Wrote inspect data to must-gather.
2026-05-27T18:33:49.824242433Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-27T18:33:49.969011162Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-27T18:33:50.275127365Z ERROR: No running kube-apiserver pods found
2026-05-27T18:33:50.308971126Z INFO: 'previous-logs' folder not found on ip-10-0-137-173.ec2.internal, skipping...
2026-05-27T18:33:50.353198354Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:33:50.384209047Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-27T18:33:50.400939674Z tar: Removing leading `/' from member names
2026-05-27T18:33:50.408564047Z INFO: HAProxy config collection complete.
2026-05-27T18:33:50.451911817Z No resources found
2026-05-27T18:33:50.459740066Z Gathering data for ns/kuadrant-system...
2026-05-27T18:33:50.526887821Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-27T18:33:50.526887821Z See 'oc rsync -h' for help and examples
2026-05-27T18:33:50.591753751Z Wrote inspect data to must-gather.
2026-05-27T18:33:50.599878230Z Wrote inspect data to must-gather.
2026-05-27T18:33:50.648954560Z Wrote inspect data to must-gather.
2026-05-27T18:33:50.654611630Z Wrote inspect data to must-gather.
2026-05-27T18:33:50.672253528Z Wrote inspect data to must-gather.
2026-05-27T18:33:50.866929758Z error: the server doesn't have a resource type "machineconfigs"
2026-05-27T18:33:50.870221150Z INFO: 'previous-logs' folder not found on ip-10-0-139-141.ec2.internal, skipping...
2026-05-27T18:33:50.995101802Z INFO: Getting rules from prometheus-k8s-0
2026-05-27T18:33:51.036017218Z Wrote inspect data to must-gather.
2026-05-27T18:33:51.213621212Z Wrote inspect data to must-gather.
2026-05-27T18:33:51.227273410Z INFO: 'previous-logs' folder not found on ip-10-0-141-157.ec2.internal, skipping...
2026-05-27T18:33:51.227310784Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-27T18:33:51.227435263Z INFO: Machine Config Daemon termination log collection complete.
2026-05-27T18:33:51.399653823Z Wrote inspect data to must-gather.
2026-05-27T18:33:51.499541922Z Wrote inspect data to must-gather.
2026-05-27T18:33:51.519158570Z INFO: Getting status/config from prometheus-k8s-0
2026-05-27T18:33:51.722525592Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-27T18:33:51.744390394Z Wrote inspect data to must-gather.
2026-05-27T18:33:51.956560000Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-27T18:33:52.022944114Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-27T18:33:52.031265015Z Wrote inspect data to must-gather.
2026-05-27T18:33:52.105058594Z INFO: INTERCONNECT MODE
2026-05-27T18:33:52.105180126Z INFO: Gathering ovn-kubernetes DBs
2026-05-27T18:33:52.352269900Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-27T18:33:52.369414343Z Wrote inspect data to must-gather.
2026-05-27T18:33:52.377750288Z INFO: Gathering OVN_Northbound from ovnkube-node-l8z5g...
2026-05-27T18:33:52.378067596Z INFO: Gathering OVN_Northbound from ovnkube-node-mvnx4...
2026-05-27T18:33:52.383155276Z INFO: Gathering OVN_Northbound from ovnkube-node-w47xc...
2026-05-27T18:33:52.383455162Z INFO: Gathering OVN_Southbound from ovnkube-node-l8z5g...
2026-05-27T18:33:52.383720320Z INFO: Gathering OVN_Southbound from ovnkube-node-mvnx4...
2026-05-27T18:33:52.384011905Z INFO: Gathering OVN_Southbound from ovnkube-node-w47xc...
2026-05-27T18:33:52.615502173Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-27T18:33:52.765978353Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-27T18:33:52.781954413Z tar: Removing leading `/' from member names
2026-05-27T18:33:52.867414176Z tar: Removing leading `/' from member names
2026-05-27T18:33:52.898494936Z tar: Removing leading `/' from member names
2026-05-27T18:33:52.902993840Z tar: Removing leading `/' from member names
2026-05-27T18:33:52.941324569Z tar: Removing leading `/' from member names
2026-05-27T18:33:52.956775413Z tar: Removing leading `/' from member names
2026-05-27T18:33:53.003635110Z Gathering data for ns/openshift-monitoring...
2026-05-27T18:33:53.105145599Z Wrote inspect data to must-gather.
2026-05-27T18:33:53.248543876Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-27T18:33:53.460476689Z Wrote inspect data to must-gather.
2026-05-27T18:33:53.564270328Z INFO: Waiting for network log collection to complete ...
2026-05-27T18:33:53.564299781Z INFO: Waiting for ovnk database copies to complete ...
2026-05-27T18:33:53.564664686Z INFO: Copying ovnk databases complete.
2026-05-27T18:33:53.566279000Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-05-27T18:33:53.570873631Z ovnk_database_store/
2026-05-27T18:33:53.570926280Z ovnk_database_store/ovnkube-node-w47xc_sbdb
2026-05-27T18:33:53.602821030Z ovnk_database_store/ovnkube-node-mvnx4_sbdb
2026-05-27T18:33:53.631289570Z ovnk_database_store/ovnkube-node-l8z5g_nbdb
2026-05-27T18:33:53.643510735Z ovnk_database_store/ovnkube-node-mvnx4_nbdb
2026-05-27T18:33:53.657036409Z ovnk_database_store/ovnkube-node-l8z5g_sbdb
2026-05-27T18:33:53.686346799Z ovnk_database_store/ovnkube-node-w47xc_nbdb
2026-05-27T18:33:53.704387418Z INFO: Network log collection complete.
2026-05-27T18:33:53.713934950Z INFO: Getting status from alertmanager-main-0
2026-05-27T18:33:53.818294077Z Wrote inspect data to must-gather.
2026-05-27T18:33:54.100835216Z Wrote inspect data to must-gather.
2026-05-27T18:33:54.355626057Z Wrote inspect data to must-gather.
2026-05-27T18:33:54.641978579Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-27T18:33:54.644372091Z INFO: Fallback to identify the container image from release info
2026-05-27T18:33:55.284904555Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-27T18:33:55.378836877Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:33:55.420731648Z daemonset.apps/perf-node-gather-daemonset created
2026-05-27T18:33:55.578074966Z Waiting for performance profile collector pods to become ready: 1
2026-05-27T18:33:55.602258071Z Gathering data for ns/openshift-network-console...
2026-05-27T18:33:56.148971968Z Gathering data for ns/openshift-console-operator...
2026-05-27T18:33:56.598948115Z Gathering data for ns/openshift-console...
2026-05-27T18:33:56.711450083Z Waiting for performance profile collector pods to become ready: 2
2026-05-27T18:33:57.197930472Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-27T18:33:57.391676239Z Wrote inspect data to must-gather.
2026-05-27T18:33:57.633731444Z Gathering data for ns/openshift-dns-operator...
2026-05-27T18:33:57.833461988Z Waiting for performance profile collector pods to become ready: 3
2026-05-27T18:33:57.990150081Z Gathering data for ns/openshift-dns...
2026-05-27T18:33:58.604906542Z Gathering data for ns/openshift-image-registry...
2026-05-27T18:33:58.953402741Z Waiting for performance profile collector pods to become ready: 4
2026-05-27T18:33:59.142877388Z Gathering data for ns/openshift-ingress-operator...
2026-05-27T18:33:59.590240605Z Gathering data for ns/openshift-ingress...
2026-05-27T18:34:00.088220414Z Waiting for performance profile collector pods to become ready: 5
2026-05-27T18:34:00.384693505Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:34:00.439607229Z Gathering data for ns/openshift-ingress-canary...
2026-05-27T18:34:00.899630719Z Gathering data for ns/openshift-insights...
2026-05-27T18:34:01.208295276Z Waiting for performance profile collector pods to become ready: 6
2026-05-27T18:34:02.315379247Z Waiting for performance profile collector pods to become ready: 7
2026-05-27T18:34:02.953739828Z Gathering data for ns/opendatahub...
2026-05-27T18:34:03.436587994Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-27T18:34:03.742464648Z Collecting performance related data for node ip-10-0-137-173.ec2.internal
2026-05-27T18:34:03.752746280Z Collecting performance related data for node ip-10-0-139-141.ec2.internal
2026-05-27T18:34:03.760580186Z Collecting performance related data for node ip-10-0-141-157.ec2.internal
2026-05-27T18:34:04.660215863Z Gathering data for ns/openshift-lws-operator...
2026-05-27T18:34:05.389614140Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:34:06.066247346Z Gathering data for ns/openshift-config...
2026-05-27T18:34:06.398342915Z Gathering data for ns/openshift-config-managed...
2026-05-27T18:34:06.940022185Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-27T18:34:07.267831008Z Gathering data for ns/openshift-kube-apiserver...
2026-05-27T18:34:07.625428101Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-27T18:34:07.971750862Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-27T18:34:08.328167793Z Gathering data for ns/openshift-kube-scheduler...
2026-05-27T18:34:08.668989946Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-27T18:34:09.061653447Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-27T18:34:09.452013831Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-27T18:34:09.817484399Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-27T18:34:10.396596235Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:34:10.412789704Z Gathering data for ns/openshift-multus...
2026-05-27T18:34:11.892976468Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-27T18:34:12.243504139Z Collecting kubelet logs for node ip-10-0-137-173.ec2.internal
2026-05-27T18:34:12.245270221Z Collecting kubelet logs for node ip-10-0-139-141.ec2.internal
2026-05-27T18:34:12.246998471Z Collecting kubelet logs for node ip-10-0-141-157.ec2.internal
2026-05-27T18:34:12.587775696Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-27T18:34:12.592394756Z INFO: Node performance data collection complete.
2026-05-27T18:34:13.628811318Z Gathering data for ns/openshift-host-network...
2026-05-27T18:34:13.996751765Z Gathering data for ns/openshift-network-diagnostics...
2026-05-27T18:34:14.590591496Z Gathering data for ns/openshift-network-node-identity...
2026-05-27T18:34:15.067991740Z Gathering data for ns/openshift-network-operator...
2026-05-27T18:34:15.402197869Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:34:15.507968022Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-27T18:34:15.794687371Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-27T18:34:16.307774662Z Gathering data for ns/openshift-apiserver-operator...
2026-05-27T18:34:16.633554699Z Gathering data for ns/openshift-apiserver...
2026-05-27T18:34:17.125432550Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-27T18:34:17.465451453Z Gathering data for ns/openshift-controller-manager...
2026-05-27T18:34:17.785490617Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-27T18:34:18.569254335Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-27T18:34:18.919119547Z Gathering data for ns/openshift-service-ca-operator...
2026-05-27T18:34:19.288199459Z Gathering data for ns/openshift-service-ca...
2026-05-27T18:34:19.690985960Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-27T18:34:20.407297313Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-27T18:34:20.750200007Z Wrote inspect data to must-gather.
2026-05-27T18:34:20.750254036Z error: inspection completed with the errors occurred while gathering data:
2026-05-27T18:34:20.750254036Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-27T18:34:20.886968515Z error: the server doesn't have a resource type "clusters"
2026-05-27T18:34:21.063186460Z Caches written to disk