2026-04-19T15:58:22.957337954Z [disk usage checker] Started
2026-04-19T15:58:22.965518064Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-19T15:58:23.194736623Z Gathering data for ns/openshift-cluster-version...
2026-04-19T15:58:23.400820349Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-19T15:58:23.487676220Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-19T15:58:23.581216857Z Gathering data for ns/default...
2026-04-19T15:58:23.924973115Z Gathering data for ns/openshift...
2026-04-19T15:58:24.367267184Z Gathering data for ns/kube-system...
2026-04-19T15:58:25.037956997Z Gathering data for ns/openshift-etcd...
2026-04-19T15:58:25.387015447Z Wrote inspect data to must-gather.
2026-04-19T15:58:25.387048641Z error: inspection completed with the errors occurred while gathering data:
2026-04-19T15:58:25.387048641Z     namespaces "assisted-installer" not found
2026-04-19T15:58:25.457810503Z Waiting on subprocesses to finish execution.
2026-04-19T15:58:25.471201380Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-19T15:58:25.472251689Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-19T15:58:25.472542396Z INFO: Collecting host service logs for crio
2026-04-19T15:58:25.472975104Z INFO: Collecting host service logs for kubelet
2026-04-19T15:58:25.473305380Z INFO: Collecting host service logs for rpm-ostreed
2026-04-19T15:58:25.473611522Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-19T15:58:25.473970191Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-19T15:58:25.474276038Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-19T15:58:25.474602130Z INFO: Collecting host service logs for NetworkManager
2026-04-19T15:58:25.474931229Z INFO: Collecting host service logs for openvswitch
2026-04-19T15:58:25.475248387Z INFO: Collecting host service logs for ovs-configuration
2026-04-19T15:58:25.475572492Z INFO: Collecting host service logs for ovsdb-server
2026-04-19T15:58:25.475917025Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-19T15:58:25.476301163Z INFO: Waiting for worker host service log collection to complete ...
2026-04-19T15:58:25.481525300Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-19T15:58:25.486141175Z INFO: Gathering HAProxy config files
2026-04-19T15:58:25.720993955Z INFO: Waiting for node performance related collection to complete ...
2026-04-19T15:58:26.268780513Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-19T15:58:26.391028443Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-19T15:58:26.414134637Z Wrote inspect data to must-gather.
2026-04-19T15:58:26.454151350Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-19T15:58:26.468396491Z No resources found
2026-04-19T15:58:26.587193250Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-19T15:58:26.815986488Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-19T15:58:26.816032950Z INFO: on-disk MachineConfig config collection complete.
2026-04-19T15:58:26.816197488Z No resources found in openshift-etcd namespace.
2026-04-19T15:58:26.832177927Z ERROR: No running kube-apiserver pods found
2026-04-19T15:58:26.886107217Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-dbkrb
2026-04-19T15:58:26.886183374Z insights-runtime-extractor-9b56l
2026-04-19T15:58:26.886202829Z insights-runtime-extractor-mf22j
2026-04-19T15:58:26.886221943Z insights-runtime-extractor-rsvd9
2026-04-19T15:58:26.901517707Z error: the server doesn't have a resource type "performanceprofile"
2026-04-19T15:58:26.969110224Z INFO: Worker host service log collection to complete.
2026-04-19T15:58:27.160071863Z INFO: "metallb-operator" not detected. Skipping.
2026-04-19T15:58:27.161308321Z No resources found
2026-04-19T15:58:27.318334513Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-19T15:58:27.580424755Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-19T15:58:27.704938988Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-19T15:58:27.721487044Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-19T15:58:27.721487044Z See 'oc rsync -h' for help and examples
2026-04-19T15:58:27.779794166Z Wrote inspect data to must-gather.
2026-04-19T15:58:27.785135114Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-19T15:58:27.850809450Z INFO: 'previous-logs' folder not found on ip-10-0-130-182.ec2.internal, skipping...
2026-04-19T15:58:27.923855100Z Wrote inspect data to must-gather.
2026-04-19T15:58:27.956957625Z Gathering data for ns/kuadrant-system...
2026-04-19T15:58:27.959659197Z Wrote inspect data to must-gather.
2026-04-19T15:58:28.003007505Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-19T15:58:28.037219733Z Wrote inspect data to must-gather.
2026-04-19T15:58:28.063980018Z Wrote inspect data to must-gather.
2026-04-19T15:58:28.165243057Z tar: Removing leading `/' from member names
2026-04-19T15:58:28.187058794Z INFO: HAProxy config collection complete.
2026-04-19T15:58:28.299383472Z error: the server doesn't have a resource type "machineconfigs"
2026-04-19T15:58:28.333022062Z Wrote inspect data to must-gather.
2026-04-19T15:58:28.385214368Z INFO: 'previous-logs' folder not found on ip-10-0-131-48.ec2.internal, skipping...
2026-04-19T15:58:28.472032371Z INFO: Getting rules from prometheus-k8s-0
2026-04-19T15:58:28.651724285Z Wrote inspect data to must-gather.
2026-04-19T15:58:28.725794669Z INFO: 'previous-logs' folder not found on ip-10-0-133-218.ec2.internal, skipping...
2026-04-19T15:58:28.725829094Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-19T15:58:28.725839850Z INFO: Machine Config Daemon termination log collection complete.
2026-04-19T15:58:28.727660758Z Wrote inspect data to must-gather.
2026-04-19T15:58:28.987414224Z Wrote inspect data to must-gather.
2026-04-19T15:58:29.008763069Z INFO: Getting status/config from prometheus-k8s-0
2026-04-19T15:58:29.064252911Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-19T15:58:29.127777599Z Wrote inspect data to must-gather.
2026-04-19T15:58:29.202299694Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-19T15:58:29.498353377Z W0419 15:58:29.498306    1134 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-19T15:58:29.518811270Z Wrote inspect data to must-gather.
2026-04-19T15:58:29.519081785Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-19T15:58:29.528907936Z INFO: INTERCONNECT MODE
2026-04-19T15:58:29.528945527Z INFO: Gathering ovn-kubernetes DBs
2026-04-19T15:58:29.622207136Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-19T15:58:29.758483456Z INFO: Gathering OVN_Northbound from ovnkube-node-kqc9s...
2026-04-19T15:58:29.758787047Z INFO: Gathering OVN_Northbound from ovnkube-node-pwk8b...
2026-04-19T15:58:29.759075333Z INFO: Gathering OVN_Northbound from ovnkube-node-xxqlx...
2026-04-19T15:58:29.759418526Z INFO: Gathering OVN_Southbound from ovnkube-node-kqc9s...
2026-04-19T15:58:29.759726892Z INFO: Gathering OVN_Southbound from ovnkube-node-pwk8b...
2026-04-19T15:58:29.760011452Z INFO: Gathering OVN_Southbound from ovnkube-node-xxqlx...
2026-04-19T15:58:29.879972448Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-19T15:58:30.008427853Z Wrote inspect data to must-gather.
2026-04-19T15:58:30.013060873Z Gathering data for ns/openshift-monitoring...
2026-04-19T15:58:30.121723952Z tar: Removing leading `/' from member names
2026-04-19T15:58:30.178389461Z tar: Removing leading `/' from member names
2026-04-19T15:58:30.186335125Z tar: Removing leading `/' from member names
2026-04-19T15:58:30.199341440Z tar: Removing leading `/' from member names
2026-04-19T15:58:30.232512414Z tar: Removing leading `/' from member names
2026-04-19T15:58:30.237937982Z tar: Removing leading `/' from member names
2026-04-19T15:58:30.306512028Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-19T15:58:30.513062439Z Wrote inspect data to must-gather.
2026-04-19T15:58:30.623458883Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-19T15:58:30.858014709Z INFO: Waiting for network log collection to complete ...
2026-04-19T15:58:30.858049408Z INFO: Waiting for ovnk database copies to complete ...
2026-04-19T15:58:30.858407376Z INFO: Copying ovnk databases complete.
2026-04-19T15:58:30.860230473Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-19T15:58:30.862946197Z ovnk_database_store/
2026-04-19T15:58:30.862946197Z Wrote inspect data to must-gather.
2026-04-19T15:58:30.862998127Z ovnk_database_store/ovnkube-node-pwk8b_sbdb
2026-04-19T15:58:30.891596674Z ovnk_database_store/ovnkube-node-kqc9s_sbdb
2026-04-19T15:58:30.921045272Z ovnk_database_store/ovnkube-node-xxqlx_sbdb
2026-04-19T15:58:30.957908576Z ovnk_database_store/ovnkube-node-pwk8b_nbdb
2026-04-19T15:58:30.970398026Z ovnk_database_store/ovnkube-node-kqc9s_nbdb
2026-04-19T15:58:30.982860395Z ovnk_database_store/ovnkube-node-xxqlx_nbdb
2026-04-19T15:58:30.997428966Z INFO: Network log collection complete.
2026-04-19T15:58:31.105309413Z INFO: Getting status from alertmanager-main-0
2026-04-19T15:58:31.225826697Z Wrote inspect data to must-gather.
2026-04-19T15:58:31.536542491Z Wrote inspect data to must-gather.
2026-04-19T15:58:31.876063707Z Wrote inspect data to must-gather.
2026-04-19T15:58:32.144219708Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-19T15:58:32.146524905Z INFO: Fallback to identify the container image from release info
2026-04-19T15:58:32.373089775Z Gathering data for ns/openshift-network-console...
2026-04-19T15:58:32.808986085Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-19T15:58:32.916866093Z Gathering data for ns/openshift-console-operator...
2026-04-19T15:58:32.950430502Z daemonset.apps/perf-node-gather-daemonset created
2026-04-19T15:58:33.017687301Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-19T15:58:33.128547783Z Waiting for performance profile collector pods to become ready: 1
2026-04-19T15:58:33.434471409Z Gathering data for ns/openshift-console...
2026-04-19T15:58:33.938405260Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-19T15:58:34.271513574Z Waiting for performance profile collector pods to become ready: 2
2026-04-19T15:58:34.367051133Z Gathering data for ns/openshift-dns-operator...
2026-04-19T15:58:34.722418937Z Gathering data for ns/openshift-dns...
2026-04-19T15:58:35.085543389Z Wrote inspect data to must-gather.
2026-04-19T15:58:35.358449745Z Gathering data for ns/openshift-image-registry...
2026-04-19T15:58:35.399447982Z Waiting for performance profile collector pods to become ready: 3
2026-04-19T15:58:35.817764880Z Gathering data for ns/openshift-ingress-operator...
2026-04-19T15:58:36.181777921Z Gathering data for ns/openshift-ingress...
2026-04-19T15:58:36.538586825Z Waiting for performance profile collector pods to become ready: 4
2026-04-19T15:58:36.875750746Z Gathering data for ns/openshift-ingress-canary...
2026-04-19T15:58:37.309591139Z Gathering data for ns/openshift-insights...
2026-04-19T15:58:37.667708280Z Waiting for performance profile collector pods to become ready: 5
2026-04-19T15:58:38.022875479Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-19T15:58:38.792820167Z Waiting for performance profile collector pods to become ready: 6
2026-04-19T15:58:39.238551363Z Gathering data for ns/opendatahub...
2026-04-19T15:58:39.918123814Z Waiting for performance profile collector pods to become ready: 7
2026-04-19T15:58:40.591671676Z Gathering data for ns/openshift-lws-operator...
2026-04-19T15:58:41.037135919Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-19T15:58:41.278908580Z Collecting performance related data for node ip-10-0-133-218.ec2.internal
2026-04-19T15:58:41.286074317Z Collecting performance related data for node ip-10-0-131-48.ec2.internal
2026-04-19T15:58:41.293315330Z Collecting performance related data for node ip-10-0-130-182.ec2.internal
2026-04-19T15:58:42.010281542Z Gathering data for ns/openshift-config...
2026-04-19T15:58:42.416028187Z Gathering data for ns/openshift-config-managed...
2026-04-19T15:58:42.972379940Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-19T15:58:43.029699634Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-19T15:58:43.267415998Z Gathering data for ns/openshift-kube-apiserver...
2026-04-19T15:58:43.596891634Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-19T15:58:44.020679277Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-19T15:58:44.381611374Z Gathering data for ns/openshift-kube-scheduler...
2026-04-19T15:58:44.683679316Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-19T15:58:45.007714144Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-19T15:58:45.353915208Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-19T15:58:45.695152842Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-19T15:58:46.345378985Z Gathering data for ns/openshift-multus...
2026-04-19T15:58:47.832826467Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-19T15:58:48.035936100Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-19T15:58:49.337099080Z Gathering data for ns/openshift-host-network...
2026-04-19T15:58:49.631262026Z Collecting kubelet logs for node ip-10-0-133-218.ec2.internal
2026-04-19T15:58:49.633445639Z Collecting kubelet logs for node ip-10-0-131-48.ec2.internal
2026-04-19T15:58:49.636207274Z Collecting kubelet logs for node ip-10-0-130-182.ec2.internal
2026-04-19T15:58:49.725387926Z Gathering data for ns/openshift-network-diagnostics...
2026-04-19T15:58:50.009097302Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-19T15:58:50.014465203Z INFO: Node performance data collection complete.
2026-04-19T15:58:50.335748423Z Gathering data for ns/openshift-network-node-identity...
2026-04-19T15:58:50.784232089Z Gathering data for ns/openshift-network-operator...
2026-04-19T15:58:51.161507413Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-19T15:58:51.462323839Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-19T15:58:51.956078209Z Gathering data for ns/openshift-apiserver-operator...
2026-04-19T15:58:52.321468025Z Gathering data for ns/openshift-apiserver...
2026-04-19T15:58:52.914696445Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-19T15:58:53.040548402Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-19T15:58:53.234116127Z Gathering data for ns/openshift-controller-manager...
2026-04-19T15:58:53.583671403Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-19T15:58:54.231374424Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-19T15:58:54.581141669Z Gathering data for ns/openshift-service-ca-operator...
2026-04-19T15:58:54.936376791Z Gathering data for ns/openshift-service-ca...
2026-04-19T15:58:55.425911756Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-19T15:58:56.395662573Z Wrote inspect data to must-gather.
2026-04-19T15:58:56.395706002Z error: inspection completed with the errors occurred while gathering data:
2026-04-19T15:58:56.395706002Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-19T15:58:56.532217658Z error: the server doesn't have a resource type "clusters"
2026-04-19T15:58:56.774760770Z Caches written to disk