2026-05-28T01:25:04.815645250Z [disk usage checker] Started
2026-05-28T01:25:04.819188358Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T01:25:05.008791203Z Gathering data for ns/openshift-cluster-version...
2026-05-28T01:25:05.132684560Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T01:25:05.218045336Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T01:25:05.292960773Z Gathering data for ns/default...
2026-05-28T01:25:05.587459036Z Gathering data for ns/openshift...
2026-05-28T01:25:05.989123929Z Gathering data for ns/kube-system...
2026-05-28T01:25:06.687263199Z Gathering data for ns/openshift-etcd...
2026-05-28T01:25:06.979996504Z Wrote inspect data to must-gather.
2026-05-28T01:25:06.980029526Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T01:25:06.980029526Z     namespaces "assisted-installer" not found
2026-05-28T01:25:07.101199582Z Waiting on subprocesses to finish execution.
2026-05-28T01:25:07.109202367Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T01:25:07.111174425Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T01:25:07.116216074Z INFO: Gathering HAProxy config files
2026-05-28T01:25:07.117148457Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T01:25:07.131052482Z INFO: Collecting host service logs for crio
2026-05-28T01:25:07.131553590Z INFO: Collecting host service logs for kubelet
2026-05-28T01:25:07.131962510Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T01:25:07.132297408Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T01:25:07.132646372Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T01:25:07.132967087Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T01:25:07.133327063Z INFO: Collecting host service logs for NetworkManager
2026-05-28T01:25:07.133721874Z INFO: Collecting host service logs for openvswitch
2026-05-28T01:25:07.134052519Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T01:25:07.134372984Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T01:25:07.134731078Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T01:25:07.135142136Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T01:25:07.182573437Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T01:25:07.791920793Z No resources found in openshift-etcd namespace.
2026-05-28T01:25:07.841764112Z Wrote inspect data to must-gather.
2026-05-28T01:25:07.925626039Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T01:25:07.996114873Z No resources found
2026-05-28T01:25:08.028553558Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T01:25:08.028657577Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T01:25:08.295598925Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T01:25:08.330086744Z INFO: Worker host service log collection to complete.
2026-05-28T01:25:08.336552293Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-qk8lr
2026-05-28T01:25:08.336596539Z insights-runtime-extractor-4flsv
2026-05-28T01:25:08.336617391Z insights-runtime-extractor-kpnsz
2026-05-28T01:25:08.336641438Z insights-runtime-extractor-tvpvm
2026-05-28T01:25:08.358895245Z Wrote inspect data to must-gather.
2026-05-28T01:25:08.376278070Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T01:25:08.412983190Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T01:25:08.434746090Z ERROR: No running kube-apiserver pods found
2026-05-28T01:25:08.441277278Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T01:25:08.459940999Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T01:25:08.849878601Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T01:25:08.966507969Z No resources found
2026-05-28T01:25:09.170075701Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T01:25:09.170075701Z See 'oc rsync -h' for help and examples
2026-05-28T01:25:09.235403344Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.242427764Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.253445378Z INFO: 'previous-logs' folder not found on ip-10-0-131-125.ec2.internal, skipping...
2026-05-28T01:25:09.263080104Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.267635788Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T01:25:09.307533484Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T01:25:09.312797590Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.352598176Z Gathering data for ns/openshift-monitoring...
2026-05-28T01:25:09.352919115Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.365041773Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T01:25:09.439731026Z tar: Removing leading `/' from member names
2026-05-28T01:25:09.444617040Z INFO: HAProxy config collection complete.
2026-05-28T01:25:09.543605348Z INFO: 'previous-logs' folder not found on ip-10-0-131-73.ec2.internal, skipping...
2026-05-28T01:25:09.543634020Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T01:25:09.574734846Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.761528977Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.789206746Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T01:25:09.802887355Z INFO: 'previous-logs' folder not found on ip-10-0-134-239.ec2.internal, skipping...
2026-05-28T01:25:09.802921924Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T01:25:09.802921924Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T01:25:09.824849558Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T01:25:09.837429416Z Wrote inspect data to must-gather.
2026-05-28T01:25:09.959891102Z Wrote inspect data to must-gather.
2026-05-28T01:25:10.066716425Z Wrote inspect data to must-gather.
2026-05-28T01:25:10.106453463Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T01:25:10.136230529Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T01:25:10.307179811Z Wrote inspect data to must-gather.
2026-05-28T01:25:10.342675418Z INFO: INTERCONNECT MODE
2026-05-28T01:25:10.342728913Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T01:25:10.491844193Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T01:25:10.496109643Z INFO: Gathering OVN_Northbound from ovnkube-node-9z98m...
2026-05-28T01:25:10.496469477Z INFO: Gathering OVN_Northbound from ovnkube-node-lfl88...
2026-05-28T01:25:10.496797831Z INFO: Gathering OVN_Northbound from ovnkube-node-wdkhx...
2026-05-28T01:25:10.497094604Z INFO: Gathering OVN_Southbound from ovnkube-node-9z98m...
2026-05-28T01:25:10.497507701Z INFO: Gathering OVN_Southbound from ovnkube-node-lfl88...
2026-05-28T01:25:10.497875854Z INFO: Gathering OVN_Southbound from ovnkube-node-wdkhx...
2026-05-28T01:25:10.533753606Z Wrote inspect data to must-gather.
2026-05-28T01:25:10.793058970Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T01:25:10.819335597Z tar: Removing leading `/' from member names
2026-05-28T01:25:10.836295526Z tar: Removing leading `/' from member names
2026-05-28T01:25:10.879834326Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T01:25:10.891962461Z tar: Removing leading `/' from member names
2026-05-28T01:25:10.893169492Z tar: Removing leading `/' from member names
2026-05-28T01:25:10.904065478Z tar: Removing leading `/' from member names
2026-05-28T01:25:10.904615508Z tar: Removing leading `/' from member names
2026-05-28T01:25:10.927185162Z Wrote inspect data to must-gather.
2026-05-28T01:25:11.121697481Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T01:25:11.159553614Z Wrote inspect data to must-gather.
2026-05-28T01:25:11.327366073Z INFO: Waiting for network log collection to complete ...
2026-05-28T01:25:11.327406233Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T01:25:11.327779563Z INFO: Copying ovnk databases complete.
2026-05-28T01:25:11.329638562Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-05-28T01:25:11.334239734Z ovnk_database_store/
2026-05-28T01:25:11.334239734Z ovnk_database_store/ovnkube-node-lfl88_nbdb
2026-05-28T01:25:11.346428684Z ovnk_database_store/ovnkube-node-lfl88_sbdb
2026-05-28T01:25:11.370515658Z W0528 01:25:11.370470    1336 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-28T01:25:11.373067928Z ovnk_database_store/ovnkube-node-9z98m_nbdb
2026-05-28T01:25:11.378073160Z Wrote inspect data to must-gather.
2026-05-28T01:25:11.384549255Z ovnk_database_store/ovnkube-node-9z98m_sbdb
2026-05-28T01:25:11.386124349Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T01:25:11.408797320Z ovnk_database_store/ovnkube-node-wdkhx_nbdb
2026-05-28T01:25:11.419223982Z ovnk_database_store/ovnkube-node-wdkhx_sbdb
2026-05-28T01:25:11.435134967Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T01:25:11.442474467Z INFO: Network log collection complete.
2026-05-28T01:25:11.607440352Z Gathering data for ns/openshift-network-console...
2026-05-28T01:25:11.695025422Z Wrote inspect data to must-gather.
2026-05-28T01:25:11.760366809Z INFO: Getting status from alertmanager-main-0
2026-05-28T01:25:11.983710473Z Wrote inspect data to must-gather.
2026-05-28T01:25:12.024588540Z Gathering data for ns/openshift-console-operator...
2026-05-28T01:25:12.227433057Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T01:25:12.229891418Z INFO: Fallback to identify the container image from release info
2026-05-28T01:25:12.383735930Z Gathering data for ns/openshift-console...
2026-05-28T01:25:12.782676816Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T01:25:12.877110876Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T01:25:12.986867221Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T01:25:13.115747200Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T01:25:13.143480308Z Gathering data for ns/openshift-dns-operator...
2026-05-28T01:25:13.480123499Z Gathering data for ns/openshift-dns...
2026-05-28T01:25:14.038510593Z Gathering data for ns/openshift-image-registry...
2026-05-28T01:25:14.237476337Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T01:25:14.496545558Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T01:25:14.796032756Z Gathering data for ns/openshift-ingress...
2026-05-28T01:25:14.829541713Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T01:25:15.127057580Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T01:25:15.349142806Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T01:25:15.498817611Z Gathering data for ns/openshift-insights...
2026-05-28T01:25:16.458375404Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T01:25:17.397768691Z Gathering data for ns/openshift-config...
2026-05-28T01:25:17.579241534Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T01:25:17.661762158Z Gathering data for ns/openshift-config-managed...
2026-05-28T01:25:18.054520747Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T01:25:18.296060875Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T01:25:18.540132057Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T01:25:18.688104847Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T01:25:18.808894601Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T01:25:19.087251467Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T01:25:19.330147611Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T01:25:19.592137065Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T01:25:19.801085066Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T01:25:19.834822479Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T01:25:19.907439091Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T01:25:20.191649927Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T01:25:20.692672048Z Gathering data for ns/openshift-multus...
2026-05-28T01:25:20.903247186Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T01:25:21.125156272Z Collecting performance related data for node ip-10-0-134-239.ec2.internal
2026-05-28T01:25:21.132121679Z Collecting performance related data for node ip-10-0-131-125.ec2.internal
2026-05-28T01:25:21.139854836Z Collecting performance related data for node ip-10-0-131-73.ec2.internal
2026-05-28T01:25:22.154809190Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T01:25:23.711836376Z Gathering data for ns/openshift-host-network...
2026-05-28T01:25:24.056742484Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T01:25:24.569147215Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T01:25:24.840703904Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T01:25:24.955427844Z Gathering data for ns/openshift-network-operator...
2026-05-28T01:25:25.319319106Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T01:25:25.580114894Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T01:25:26.025856940Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T01:25:26.324220021Z Gathering data for ns/openshift-apiserver...
2026-05-28T01:25:26.698659012Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T01:25:26.947498725Z Gathering data for ns/openshift-controller-manager...
2026-05-28T01:25:27.210821142Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T01:25:27.844781059Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T01:25:28.137355657Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T01:25:28.455144889Z Gathering data for ns/openshift-service-ca...
2026-05-28T01:25:28.814227543Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T01:25:29.096451819Z Collecting kubelet logs for node ip-10-0-134-239.ec2.internal
2026-05-28T01:25:29.098150359Z Collecting kubelet logs for node ip-10-0-131-125.ec2.internal
2026-05-28T01:25:29.099806420Z Collecting kubelet logs for node ip-10-0-131-73.ec2.internal
2026-05-28T01:25:29.462667099Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T01:25:29.468528501Z INFO: Node performance data collection complete.
2026-05-28T01:25:29.721465402Z Wrote inspect data to must-gather.
2026-05-28T01:25:29.721504046Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T01:25:29.721504046Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T01:25:29.848634748Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T01:25:29.852872829Z error: the server doesn't have a resource type "clusters"
2026-05-28T01:25:29.984793881Z Caches written to disk