2026-05-29T01:22:53.443746348Z [disk usage checker] Started
2026-05-29T01:22:53.447135685Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-29T01:22:53.659750513Z Gathering data for ns/openshift-cluster-version...
2026-05-29T01:22:53.837256130Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-29T01:22:53.932095719Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-29T01:22:54.041370461Z Gathering data for ns/default...
2026-05-29T01:22:54.376708919Z Gathering data for ns/openshift...
2026-05-29T01:22:54.822977678Z Gathering data for ns/kube-system...
2026-05-29T01:22:55.502910016Z Gathering data for ns/openshift-etcd...
2026-05-29T01:22:55.912804942Z Waiting on subprocesses to finish execution.
2026-05-29T01:22:55.917026333Z INFO: Gathering HAProxy config files
2026-05-29T01:22:55.919704039Z INFO: Collecting host service logs for crio
2026-05-29T01:22:55.919994252Z INFO: Collecting host service logs for kubelet
2026-05-29T01:22:55.920332807Z INFO: Collecting host service logs for rpm-ostreed
2026-05-29T01:22:55.920631015Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-29T01:22:55.920956326Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-29T01:22:55.921350371Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-29T01:22:55.921886454Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-29T01:22:55.921945709Z INFO: Collecting host service logs for NetworkManager
2026-05-29T01:22:55.922261300Z INFO: Collecting host service logs for openvswitch
2026-05-29T01:22:55.922576095Z INFO: Collecting host service logs for ovs-configuration
2026-05-29T01:22:55.922935775Z INFO: Collecting host service logs for ovsdb-server
2026-05-29T01:22:55.923269228Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-29T01:22:55.923646019Z INFO: Waiting for worker host service log collection to complete ...
2026-05-29T01:22:55.935256793Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-29T01:22:55.964070132Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-29T01:22:56.037008002Z INFO: Waiting for node performance related collection to complete ...
2026-05-29T01:22:56.174722503Z Wrote inspect data to must-gather.
2026-05-29T01:22:56.174755953Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-29T01:22:56.174811065Z INFO: on-disk MachineConfig config collection complete.
2026-05-29T01:22:56.177299271Z error: inspection completed with the errors occurred while gathering data:
2026-05-29T01:22:56.177299271Z     namespaces "assisted-installer" not found
2026-05-29T01:22:56.783095616Z error: the server doesn't have a resource type "performanceprofile"
2026-05-29T01:22:56.820190983Z No resources found in openshift-etcd namespace.
2026-05-29T01:22:57.012019987Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-29T01:22:57.091698740Z INFO: "metallb-operator" not detected. Skipping.
2026-05-29T01:22:57.275234780Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-29T01:22:57.334855690Z INFO: Worker host service log collection to complete.
2026-05-29T01:22:57.382917775Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-66nzn
2026-05-29T01:22:57.382917775Z insights-runtime-extractor-7gnqf
2026-05-29T01:22:57.382917775Z insights-runtime-extractor-bxtgb
2026-05-29T01:22:57.382917775Z insights-runtime-extractor-rn29k
2026-05-29T01:22:57.588161170Z No resources found
2026-05-29T01:22:57.627292663Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-29T01:22:57.880260347Z Wrote inspect data to must-gather.
2026-05-29T01:22:57.913128169Z Wrote inspect data to must-gather.
2026-05-29T01:22:57.951552298Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-29T01:22:57.960721631Z ERROR: No running kube-apiserver pods found
2026-05-29T01:22:57.986868773Z No resources found
2026-05-29T01:22:58.058251701Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-29T01:22:58.189153286Z Wrote inspect data to must-gather.
2026-05-29T01:22:58.290084939Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-29T01:22:58.484142506Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-29T01:22:58.484142506Z See 'oc rsync -h' for help and examples
2026-05-29T01:22:58.516992783Z Wrote inspect data to must-gather.
2026-05-29T01:22:58.552252855Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-29T01:22:58.566937208Z Wrote inspect data to must-gather.
2026-05-29T01:22:58.575845400Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-29T01:22:58.596028134Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-29T01:22:58.694520358Z error: the server doesn't have a resource type "machineconfigs"
2026-05-29T01:22:58.705843284Z INFO: 'previous-logs' folder not found on ip-10-0-131-222.ec2.internal, skipping...
2026-05-29T01:22:58.762167953Z Gathering data for ns/kuadrant-system...
2026-05-29T01:22:58.773686680Z tar: Removing leading `/' from member names
2026-05-29T01:22:58.783566279Z INFO: HAProxy config collection complete.
2026-05-29T01:22:58.789491852Z Wrote inspect data to must-gather.
2026-05-29T01:22:58.833271550Z Wrote inspect data to must-gather.
2026-05-29T01:22:59.034180970Z Wrote inspect data to must-gather.
2026-05-29T01:22:59.047871663Z INFO: Getting rules from prometheus-k8s-0
2026-05-29T01:22:59.082604519Z INFO: 'previous-logs' folder not found on ip-10-0-140-228.ec2.internal, skipping...
2026-05-29T01:22:59.158201784Z Wrote inspect data to must-gather.
2026-05-29T01:22:59.443062113Z Wrote inspect data to must-gather.
2026-05-29T01:22:59.511899243Z INFO: 'previous-logs' folder not found on ip-10-0-141-236.ec2.internal, skipping...
2026-05-29T01:22:59.511934538Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-29T01:22:59.511967825Z INFO: Machine Config Daemon termination log collection complete.
2026-05-29T01:22:59.584880341Z Wrote inspect data to must-gather.
2026-05-29T01:22:59.598042611Z INFO: Getting status/config from prometheus-k8s-0
2026-05-29T01:22:59.657938318Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-29T01:22:59.958942462Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-29T01:22:59.978712406Z INFO: INTERCONNECT MODE
2026-05-29T01:22:59.978752809Z INFO: Gathering ovn-kubernetes DBs
2026-05-29T01:22:59.983672918Z Wrote inspect data to must-gather.
2026-05-29T01:23:00.080421011Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-29T01:23:00.180753259Z INFO: Gathering OVN_Northbound from ovnkube-node-2ftcp...
2026-05-29T01:23:00.181009628Z INFO: Gathering OVN_Northbound from ovnkube-node-4vn9b...
2026-05-29T01:23:00.181286639Z INFO: Gathering OVN_Northbound from ovnkube-node-hn2k9...
2026-05-29T01:23:00.181603722Z INFO: Gathering OVN_Southbound from ovnkube-node-2ftcp...
2026-05-29T01:23:00.181913195Z INFO: Gathering OVN_Southbound from ovnkube-node-4vn9b...
2026-05-29T01:23:00.182193216Z INFO: Gathering OVN_Southbound from ovnkube-node-hn2k9...
2026-05-29T01:23:00.335929276Z Wrote inspect data to must-gather.
2026-05-29T01:23:00.407837806Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-29T01:23:00.535599008Z tar: Removing leading `/' from member names
2026-05-29T01:23:00.568698170Z tar: Removing leading `/' from member names
2026-05-29T01:23:00.648825196Z tar: Removing leading `/' from member names
2026-05-29T01:23:00.670898038Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-29T01:23:00.705661149Z tar: Removing leading `/' from member names
2026-05-29T01:23:00.746681695Z tar: Removing leading `/' from member names
2026-05-29T01:23:00.749989558Z tar: Removing leading `/' from member names
2026-05-29T01:23:00.890898383Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-29T01:23:00.903344733Z Wrote inspect data to must-gather.
2026-05-29T01:23:00.906976144Z Gathering data for ns/openshift-monitoring...
2026-05-29T01:23:01.160742719Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-29T01:23:01.301463808Z Wrote inspect data to must-gather.
2026-05-29T01:23:01.348506657Z INFO: Waiting for network log collection to complete ...
2026-05-29T01:23:01.348554066Z INFO: Waiting for ovnk database copies to complete ...
2026-05-29T01:23:01.349104488Z INFO: Copying ovnk databases complete.
2026-05-29T01:23:01.350759576Z 4.4M	must-gather/network_logs/ovnk_database_store
2026-05-29T01:23:01.352985094Z ovnk_database_store/
2026-05-29T01:23:01.352985094Z ovnk_database_store/ovnkube-node-2ftcp_nbdb
2026-05-29T01:23:01.399277798Z ovnk_database_store/ovnkube-node-4vn9b_nbdb
2026-05-29T01:23:01.409485846Z ovnk_database_store/ovnkube-node-hn2k9_nbdb
2026-05-29T01:23:01.420484258Z ovnk_database_store/ovnkube-node-4vn9b_sbdb
2026-05-29T01:23:01.481762806Z ovnk_database_store/ovnkube-node-hn2k9_sbdb
2026-05-29T01:23:01.513550776Z ovnk_database_store/ovnkube-node-2ftcp_sbdb
2026-05-29T01:23:01.544242931Z INFO: Network log collection complete.
2026-05-29T01:23:01.588599644Z INFO: Getting status from alertmanager-main-0
2026-05-29T01:23:01.636295462Z Wrote inspect data to must-gather.
2026-05-29T01:23:01.969315489Z Wrote inspect data to must-gather.
2026-05-29T01:23:02.297371288Z Wrote inspect data to must-gather.
2026-05-29T01:23:02.581636091Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-29T01:23:02.584745704Z INFO: Fallback to identify the container image from release info
2026-05-29T01:23:02.926927990Z Gathering data for ns/openshift-network-console...
2026-05-29T01:23:03.266530285Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-29T01:23:03.415019480Z daemonset.apps/perf-node-gather-daemonset created
2026-05-29T01:23:03.422087598Z Gathering data for ns/openshift-console-operator...
2026-05-29T01:23:03.566712833Z Waiting for performance profile collector pods to become ready: 1
2026-05-29T01:23:03.609660430Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-29T01:23:03.928919496Z Gathering data for ns/openshift-console...
2026-05-29T01:23:04.417945154Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-29T01:23:04.693800342Z Waiting for performance profile collector pods to become ready: 2
2026-05-29T01:23:04.854368093Z Gathering data for ns/openshift-dns-operator...
2026-05-29T01:23:05.192767090Z Gathering data for ns/openshift-dns...
2026-05-29T01:23:05.671478455Z Wrote inspect data to must-gather.
2026-05-29T01:23:05.821831334Z Gathering data for ns/openshift-image-registry...
2026-05-29T01:23:05.826915993Z Waiting for performance profile collector pods to become ready: 3
2026-05-29T01:23:06.290972070Z Gathering data for ns/openshift-ingress-operator...
2026-05-29T01:23:06.626906030Z Gathering data for ns/openshift-ingress...
2026-05-29T01:23:06.949703387Z Waiting for performance profile collector pods to become ready: 4
2026-05-29T01:23:07.498826953Z Gathering data for ns/openshift-ingress-canary...
2026-05-29T01:23:07.949770824Z Gathering data for ns/openshift-insights...
2026-05-29T01:23:08.055864626Z Waiting for performance profile collector pods to become ready: 5
2026-05-29T01:23:08.614063398Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-29T01:23:09.166977723Z Waiting for performance profile collector pods to become ready: 6
2026-05-29T01:23:09.863457951Z Gathering data for ns/opendatahub...
2026-05-29T01:23:10.279580876Z Waiting for performance profile collector pods to become ready: 7
2026-05-29T01:23:11.400151186Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-29T01:23:11.457315391Z Gathering data for ns/openshift-lws-operator...
2026-05-29T01:23:11.702395491Z Collecting performance related data for node ip-10-0-140-228.ec2.internal
2026-05-29T01:23:11.713625625Z Collecting performance related data for node ip-10-0-131-222.ec2.internal
2026-05-29T01:23:11.723744912Z Collecting performance related data for node ip-10-0-141-236.ec2.internal
2026-05-29T01:23:12.914746104Z Gathering data for ns/openshift-config...
2026-05-29T01:23:13.296503575Z Gathering data for ns/openshift-config-managed...
2026-05-29T01:23:13.619576947Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-29T01:23:13.737992851Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-29T01:23:14.101323151Z Gathering data for ns/openshift-kube-apiserver...
2026-05-29T01:23:14.431972103Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-29T01:23:14.780648717Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-29T01:23:15.094085568Z Gathering data for ns/openshift-kube-scheduler...
2026-05-29T01:23:15.377461050Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-29T01:23:15.698017916Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-29T01:23:16.039737427Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-29T01:23:16.367863564Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-29T01:23:17.014727867Z Gathering data for ns/openshift-multus...
2026-05-29T01:23:18.430494701Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-29T01:23:18.625246564Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-29T01:23:19.772511624Z Collecting kubelet logs for node ip-10-0-140-228.ec2.internal
2026-05-29T01:23:19.774606065Z Collecting kubelet logs for node ip-10-0-131-222.ec2.internal
2026-05-29T01:23:19.776185810Z Collecting kubelet logs for node ip-10-0-141-236.ec2.internal
2026-05-29T01:23:20.082302920Z Gathering data for ns/openshift-host-network...
2026-05-29T01:23:20.183953332Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-29T01:23:20.189964862Z INFO: Node performance data collection complete.
2026-05-29T01:23:20.456165485Z Gathering data for ns/openshift-network-diagnostics...
2026-05-29T01:23:21.013968675Z Gathering data for ns/openshift-network-node-identity...
2026-05-29T01:23:21.457393179Z Gathering data for ns/openshift-network-operator...
2026-05-29T01:23:21.824377966Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-29T01:23:22.154254278Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-29T01:23:22.600709970Z Gathering data for ns/openshift-apiserver-operator...
2026-05-29T01:23:22.944157838Z Gathering data for ns/openshift-apiserver...
2026-05-29T01:23:23.350426803Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-29T01:23:23.630219940Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-29T01:23:23.630830349Z Gathering data for ns/openshift-controller-manager...
2026-05-29T01:23:23.933178351Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-29T01:23:24.566087128Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-29T01:23:24.937167819Z Gathering data for ns/openshift-service-ca-operator...
2026-05-29T01:23:25.249297738Z Gathering data for ns/openshift-service-ca...
2026-05-29T01:23:25.624521224Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-29T01:23:26.561810076Z Wrote inspect data to must-gather.
2026-05-29T01:23:26.561857266Z error: inspection completed with the errors occurred while gathering data:
2026-05-29T01:23:26.561857266Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-29T01:23:26.690881306Z error: the server doesn't have a resource type "clusters"
2026-05-29T01:23:27.016895755Z Caches written to disk