2026-04-24T21:53:57.892090179Z [disk usage checker] Started
2026-04-24T21:53:57.895794038Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-04-24T21:53:58.093833783Z Gathering data for ns/openshift-cluster-version...
2026-04-24T21:53:58.242945354Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T21:53:58.332832850Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T21:53:58.420463158Z Gathering data for ns/default...
2026-04-24T21:53:58.739578654Z Gathering data for ns/openshift...
2026-04-24T21:53:59.166637470Z Gathering data for ns/kube-system...
2026-04-24T21:53:59.897255794Z Gathering data for ns/openshift-etcd...
2026-04-24T21:54:00.254938263Z Wrote inspect data to must-gather.
2026-04-24T21:54:00.254971533Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T21:54:00.254971533Z     namespaces "assisted-installer" not found
2026-04-24T21:54:00.309125933Z Waiting on subprocesses to finish execution.
2026-04-24T21:54:00.312680557Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-24T21:54:00.316702077Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-24T21:54:00.317260869Z INFO: Collecting host service logs for crio
2026-04-24T21:54:00.317464103Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-24T21:54:00.317686199Z INFO: Collecting host service logs for kubelet
2026-04-24T21:54:00.317984826Z INFO: Collecting host service logs for rpm-ostreed
2026-04-24T21:54:00.318285393Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-24T21:54:00.318602809Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-24T21:54:00.318929013Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-24T21:54:00.319270382Z INFO: Collecting host service logs for NetworkManager
2026-04-24T21:54:00.319621488Z INFO: Collecting host service logs for openvswitch
2026-04-24T21:54:00.319966446Z INFO: Collecting host service logs for ovs-configuration
2026-04-24T21:54:00.320316562Z INFO: Collecting host service logs for ovsdb-server
2026-04-24T21:54:00.320682404Z INFO: Gathering HAProxy config files
2026-04-24T21:54:00.320780116Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-24T21:54:00.321216814Z INFO: Waiting for worker host service log collection to complete ...
2026-04-24T21:54:00.415038263Z INFO: Waiting for node performance related collection to complete ...
2026-04-24T21:54:01.080639289Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-24T21:54:01.087633456Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-24T21:54:01.102804733Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-24T21:54:01.102835773Z INFO: on-disk MachineConfig config collection complete.
2026-04-24T21:54:01.138716717Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-24T21:54:01.246054004Z INFO: "metallb-operator" not detected. Skipping.
2026-04-24T21:54:01.260105678Z Wrote inspect data to must-gather.
2026-04-24T21:54:01.283978186Z No resources found
2026-04-24T21:54:01.376313561Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-24T21:54:01.400416246Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-crpwz
2026-04-24T21:54:01.400416246Z insights-runtime-extractor-2bz6f
2026-04-24T21:54:01.400416246Z insights-runtime-extractor-ftrc6
2026-04-24T21:54:01.400416246Z insights-runtime-extractor-lrjh7
2026-04-24T21:54:01.436831021Z No resources found in openshift-etcd namespace.
2026-04-24T21:54:01.501299308Z INFO: Worker host service log collection to complete.
2026-04-24T21:54:01.607697884Z error: the server doesn't have a resource type "performanceprofile"
2026-04-24T21:54:01.628819756Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-24T21:54:01.628819756Z See 'oc rsync -h' for help and examples
2026-04-24T21:54:01.652763411Z ERROR: No running kube-apiserver pods found
2026-04-24T21:54:01.895216613Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-24T21:54:01.944066968Z Wrote inspect data to must-gather.
2026-04-24T21:54:02.123840112Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-24T21:54:02.180845140Z No resources found
2026-04-24T21:54:02.192165998Z Wrote inspect data to must-gather.
2026-04-24T21:54:02.207544258Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T21:54:02.394570936Z Gathering data for ns/openshift-monitoring...
2026-04-24T21:54:02.404929364Z INFO: 'previous-logs' folder not found on ip-10-0-134-249.ec2.internal, skipping...
2026-04-24T21:54:02.447681923Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-24T21:54:02.529866932Z Wrote inspect data to must-gather.
2026-04-24T21:54:02.608032797Z Wrote inspect data to must-gather.
2026-04-24T21:54:02.621974655Z Wrote inspect data to must-gather.
2026-04-24T21:54:02.657717213Z error: the server doesn't have a resource type "machineconfigs"
2026-04-24T21:54:02.683264513Z Wrote inspect data to must-gather.
2026-04-24T21:54:02.689612613Z tar: Removing leading `/' from member names
2026-04-24T21:54:02.694560127Z INFO: HAProxy config collection complete.
2026-04-24T21:54:02.754496983Z INFO: Getting rules from prometheus-k8s-0
2026-04-24T21:54:02.774589713Z INFO: 'previous-logs' folder not found on ip-10-0-135-27.ec2.internal, skipping...
2026-04-24T21:54:02.889383808Z Wrote inspect data to must-gather.
2026-04-24T21:54:02.901497070Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-04-24T21:54:02.953773492Z Wrote inspect data to must-gather.
2026-04-24T21:54:03.032693275Z INFO: 'previous-logs' folder not found on ip-10-0-142-162.ec2.internal, skipping...
2026-04-24T21:54:03.032797447Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-24T21:54:03.032851426Z INFO: Machine Config Daemon termination log collection complete.
2026-04-24T21:54:03.113634199Z Wrote inspect data to must-gather.
2026-04-24T21:54:03.190575883Z INFO: Getting status/config from prometheus-k8s-0
2026-04-24T21:54:03.195854837Z Wrote inspect data to must-gather.
2026-04-24T21:54:03.268492893Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-24T21:54:03.418833896Z Wrote inspect data to must-gather.
2026-04-24T21:54:03.484038167Z Wrote inspect data to must-gather.
2026-04-24T21:54:03.585648058Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-24T21:54:03.699627774Z Wrote inspect data to must-gather.
2026-04-24T21:54:03.705911428Z INFO: INTERCONNECT MODE
2026-04-24T21:54:03.705938957Z INFO: Gathering ovn-kubernetes DBs
2026-04-24T21:54:03.828538262Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-24T21:54:03.875816506Z INFO: Gathering OVN_Northbound from ovnkube-node-256cw...
2026-04-24T21:54:03.876075579Z INFO: Gathering OVN_Northbound from ovnkube-node-qccqq...
2026-04-24T21:54:03.876381933Z INFO: Gathering OVN_Northbound from ovnkube-node-qr6dh...
2026-04-24T21:54:03.876704460Z INFO: Gathering OVN_Southbound from ovnkube-node-256cw...
2026-04-24T21:54:03.876972437Z INFO: Gathering OVN_Southbound from ovnkube-node-qccqq...
2026-04-24T21:54:03.877239602Z INFO: Gathering OVN_Southbound from ovnkube-node-qr6dh...
2026-04-24T21:54:03.934080499Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T21:54:03.961475694Z Wrote inspect data to must-gather.
2026-04-24T21:54:04.173188162Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-24T21:54:04.183260550Z tar: Removing leading `/' from member names
2026-04-24T21:54:04.193050297Z tar: Removing leading `/' from member names
2026-04-24T21:54:04.207265485Z tar: Removing leading `/' from member names
2026-04-24T21:54:04.218835726Z tar: Removing leading `/' from member names
2026-04-24T21:54:04.223230154Z tar: Removing leading `/' from member names
2026-04-24T21:54:04.249183251Z tar: Removing leading `/' from member names
2026-04-24T21:54:04.301308987Z Wrote inspect data to must-gather.
2026-04-24T21:54:04.457784817Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-24T21:54:04.533614207Z W0424 21:54:04.533563    1305 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-24T21:54:04.539621763Z Wrote inspect data to must-gather.
2026-04-24T21:54:04.653585336Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T21:54:04.696076311Z INFO: Waiting for network log collection to complete ...
2026-04-24T21:54:04.696244775Z INFO: Waiting for ovnk database copies to complete ...
2026-04-24T21:54:04.696826994Z INFO: Copying ovnk databases complete.
2026-04-24T21:54:04.698823242Z 3.3M	must-gather/network_logs/ovnk_database_store
2026-04-24T21:54:04.702530147Z ovnk_database_store/
2026-04-24T21:54:04.702530147Z ovnk_database_store/ovnkube-node-qr6dh_nbdb
2026-04-24T21:54:04.715434005Z ovnk_database_store/ovnkube-node-256cw_sbdb
2026-04-24T21:54:04.735361396Z ovnk_database_store/ovnkube-node-qccqq_sbdb
2026-04-24T21:54:04.755808840Z ovnk_database_store/ovnkube-node-256cw_nbdb
2026-04-24T21:54:04.763670815Z ovnk_database_store/ovnkube-node-qccqq_nbdb
2026-04-24T21:54:04.771173848Z ovnk_database_store/ovnkube-node-qr6dh_sbdb
2026-04-24T21:54:04.799301985Z INFO: Network log collection complete.
2026-04-24T21:54:04.852531648Z Wrote inspect data to must-gather.
2026-04-24T21:54:04.855495227Z INFO: Getting status from alertmanager-main-0
2026-04-24T21:54:04.867627615Z Gathering data for ns/openshift-network-console...
2026-04-24T21:54:05.129443902Z Wrote inspect data to must-gather.
2026-04-24T21:54:05.307611035Z Gathering data for ns/openshift-console-operator...
2026-04-24T21:54:05.389268528Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-24T21:54:05.391746034Z INFO: Fallback to identify the container image from release info
2026-04-24T21:54:05.669749661Z Gathering data for ns/openshift-console...
2026-04-24T21:54:05.859034674Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-24T21:54:05.980692207Z daemonset.apps/perf-node-gather-daemonset created
2026-04-24T21:54:06.110940090Z Waiting for performance profile collector pods to become ready: 1
2026-04-24T21:54:06.123709989Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-24T21:54:06.503728929Z Gathering data for ns/openshift-dns-operator...
2026-04-24T21:54:06.839745124Z Gathering data for ns/openshift-dns...
2026-04-24T21:54:07.239242328Z Waiting for performance profile collector pods to become ready: 2
2026-04-24T21:54:07.553527697Z Gathering data for ns/openshift-image-registry...
2026-04-24T21:54:07.905859417Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-04-24T21:54:08.067864938Z Gathering data for ns/openshift-ingress-operator...
2026-04-24T21:54:08.378551719Z Waiting for performance profile collector pods to become ready: 3
2026-04-24T21:54:08.405569511Z Gathering data for ns/openshift-ingress...
2026-04-24T21:54:08.812459860Z Gathering data for ns/openshift-ingress-canary...
2026-04-24T21:54:09.189808639Z Gathering data for ns/openshift-insights...
2026-04-24T21:54:09.509298985Z Waiting for performance profile collector pods to become ready: 4
2026-04-24T21:54:10.636280537Z Waiting for performance profile collector pods to become ready: 5
2026-04-24T21:54:11.294547113Z Gathering data for ns/kserve...
2026-04-24T21:54:11.759636131Z Waiting for performance profile collector pods to become ready: 6
2026-04-24T21:54:12.729228802Z Gathering data for ns/openshift-config...
2026-04-24T21:54:12.877013587Z Waiting for performance profile collector pods to become ready: 7
2026-04-24T21:54:12.910232156Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-04-24T21:54:13.052585819Z Gathering data for ns/openshift-config-managed...
2026-04-24T21:54:13.514719530Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-24T21:54:13.819417834Z Gathering data for ns/openshift-kube-apiserver...
2026-04-24T21:54:13.989910130Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-24T21:54:14.107533768Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-24T21:54:14.260259862Z Collecting performance related data for node ip-10-0-134-249.ec2.internal
2026-04-24T21:54:14.267618699Z Collecting performance related data for node ip-10-0-135-27.ec2.internal
2026-04-24T21:54:14.280002288Z Collecting performance related data for node ip-10-0-142-162.ec2.internal
2026-04-24T21:54:14.461180975Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-24T21:54:14.815314832Z Gathering data for ns/openshift-kube-scheduler...
2026-04-24T21:54:15.183754029Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-24T21:54:15.573924399Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-24T21:54:15.957393026Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-24T21:54:16.277967858Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-24T21:54:16.970516655Z Gathering data for ns/openshift-multus...
2026-04-24T21:54:17.914749915Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-04-24T21:54:18.483333914Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-24T21:54:19.982110358Z Gathering data for ns/openshift-host-network...
2026-04-24T21:54:20.297194530Z Gathering data for ns/openshift-network-diagnostics...
2026-04-24T21:54:20.824661775Z Gathering data for ns/openshift-network-node-identity...
2026-04-24T21:54:21.247325186Z Gathering data for ns/openshift-network-operator...
2026-04-24T21:54:21.684789666Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-24T21:54:21.959489822Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-24T21:54:22.234130302Z Collecting kubelet logs for node ip-10-0-134-249.ec2.internal
2026-04-24T21:54:22.235876379Z Collecting kubelet logs for node ip-10-0-135-27.ec2.internal
2026-04-24T21:54:22.238049488Z Collecting kubelet logs for node ip-10-0-142-162.ec2.internal
2026-04-24T21:54:22.488946411Z Gathering data for ns/openshift-apiserver-operator...
2026-04-24T21:54:22.615931043Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-24T21:54:22.620920322Z INFO: Node performance data collection complete.
2026-04-24T21:54:22.818867961Z Gathering data for ns/openshift-apiserver...
2026-04-24T21:54:22.919700806Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-04-24T21:54:23.225483382Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-24T21:54:23.533439862Z Gathering data for ns/openshift-controller-manager...
2026-04-24T21:54:23.813927507Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-24T21:54:24.447210335Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-24T21:54:24.790311708Z Gathering data for ns/openshift-service-ca-operator...
2026-04-24T21:54:25.103587174Z Gathering data for ns/openshift-service-ca...
2026-04-24T21:54:25.509714349Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-24T21:54:26.406368672Z Wrote inspect data to must-gather.
2026-04-24T21:54:26.406410568Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T21:54:26.406410568Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-24T21:54:26.539841613Z error: the server doesn't have a resource type "clusters"
2026-04-24T21:54:26.678236127Z Caches written to disk