2026-05-21T02:21:57.046761453Z [disk usage checker] Started
2026-05-21T02:21:57.050835937Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T02:21:57.267404793Z Gathering data for ns/openshift-cluster-version...
2026-05-21T02:21:57.414238601Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T02:21:57.503196963Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T02:21:57.597429451Z Gathering data for ns/default...
2026-05-21T02:21:57.916136295Z Gathering data for ns/openshift...
2026-05-21T02:21:58.361578874Z Gathering data for ns/kube-system...
2026-05-21T02:21:59.088207554Z Gathering data for ns/openshift-etcd...
2026-05-21T02:21:59.406118841Z Waiting on subprocesses to finish execution.
2026-05-21T02:21:59.408311886Z INFO: Gathering HAProxy config files
2026-05-21T02:21:59.432495505Z Wrote inspect data to must-gather.
2026-05-21T02:21:59.432593893Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T02:21:59.432593893Z     namespaces "assisted-installer" not found
2026-05-21T02:21:59.440562193Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-21T02:21:59.444856172Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-21T02:21:59.448366711Z INFO: Waiting for node performance related collection to complete ...
2026-05-21T02:21:59.454516197Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-21T02:21:59.493141738Z INFO: Collecting host service logs for crio
2026-05-21T02:21:59.493440232Z INFO: Collecting host service logs for kubelet
2026-05-21T02:21:59.493806732Z INFO: Collecting host service logs for rpm-ostreed
2026-05-21T02:21:59.494142420Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-21T02:21:59.494498499Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-21T02:21:59.494847257Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-21T02:21:59.495164960Z INFO: Collecting host service logs for NetworkManager
2026-05-21T02:21:59.495492330Z INFO: Collecting host service logs for openvswitch
2026-05-21T02:21:59.495816075Z INFO: Collecting host service logs for ovs-configuration
2026-05-21T02:21:59.496165815Z INFO: Collecting host service logs for ovsdb-server
2026-05-21T02:21:59.496527002Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-21T02:21:59.496924451Z INFO: Waiting for worker host service log collection to complete ...
2026-05-21T02:21:59.942887433Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-g7xz9
2026-05-21T02:21:59.943107874Z insights-runtime-extractor-4rtfw
2026-05-21T02:21:59.943135903Z insights-runtime-extractor-n8lc5
2026-05-21T02:21:59.943158827Z insights-runtime-extractor-sqcgc
2026-05-21T02:21:59.973893272Z No resources found in openshift-etcd namespace.
2026-05-21T02:22:00.009558942Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-21T02:22:00.023605857Z No resources found
2026-05-21T02:22:00.143652727Z error: the server doesn't have a resource type "performanceprofile"
2026-05-21T02:22:00.160199883Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-21T02:22:00.315200498Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-21T02:22:00.315377213Z INFO: on-disk MachineConfig config collection complete.
2026-05-21T02:22:00.335416510Z Wrote inspect data to must-gather.
2026-05-21T02:22:00.439863469Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-21T02:22:00.472176713Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-21T02:22:00.472176713Z See 'oc rsync -h' for help and examples
2026-05-21T02:22:00.586926380Z ERROR: No running kube-apiserver pods found
2026-05-21T02:22:00.596218085Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-21T02:22:00.826902027Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-21T02:22:00.848428023Z INFO: 'previous-logs' folder not found on ip-10-0-129-167.ec2.internal, skipping...
2026-05-21T02:22:01.068811056Z INFO: "metallb-operator" not detected. Skipping.
2026-05-21T02:22:01.189206653Z tar: Removing leading `/' from member names
2026-05-21T02:22:01.194917206Z INFO: HAProxy config collection complete.
2026-05-21T02:22:01.217874672Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-21T02:22:01.223683208Z INFO: Worker host service log collection to complete.
2026-05-21T02:22:01.428088054Z Wrote inspect data to must-gather.
2026-05-21T02:22:01.943956121Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-21T02:22:02.029673191Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T02:22:02.055884160Z No resources found
2026-05-21T02:22:02.112302368Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T02:22:02.119628885Z Gathering data for ns/kuadrant-system...
2026-05-21T02:22:02.233992307Z INFO: 'previous-logs' folder not found on ip-10-0-136-250.ec2.internal, skipping...
2026-05-21T02:22:02.242304183Z Wrote inspect data to must-gather.
2026-05-21T02:22:02.295796238Z error: the server doesn't have a resource type "machineconfigs"
2026-05-21T02:22:02.313770557Z Wrote inspect data to must-gather.
2026-05-21T02:22:02.351929576Z Wrote inspect data to must-gather.
2026-05-21T02:22:02.371063473Z Wrote inspect data to must-gather.
2026-05-21T02:22:02.488537488Z INFO: Getting rules from prometheus-k8s-0
2026-05-21T02:22:02.696857177Z INFO: 'previous-logs' folder not found on ip-10-0-141-162.ec2.internal, skipping...
2026-05-21T02:22:02.696969399Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-21T02:22:02.697024545Z INFO: Machine Config Daemon termination log collection complete.
2026-05-21T02:22:02.698186310Z Wrote inspect data to must-gather.
2026-05-21T02:22:02.786224385Z Wrote inspect data to must-gather.
2026-05-21T02:22:03.037444989Z INFO: Getting status/config from prometheus-k8s-0
2026-05-21T02:22:03.038961734Z Wrote inspect data to must-gather.
2026-05-21T02:22:03.187360183Z Wrote inspect data to must-gather.
2026-05-21T02:22:03.253825218Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-21T02:22:03.262477609Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T02:22:03.519137208Z Wrote inspect data to must-gather.
2026-05-21T02:22:03.539805129Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-21T02:22:03.574495452Z Wrote inspect data to must-gather.
2026-05-21T02:22:03.843216416Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-21T02:22:03.852433917Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T02:22:03.906608691Z INFO: INTERCONNECT MODE
2026-05-21T02:22:03.906713723Z INFO: Gathering ovn-kubernetes DBs
2026-05-21T02:22:04.013978636Z Wrote inspect data to must-gather.
2026-05-21T02:22:04.089869169Z Gathering data for ns/openshift-monitoring...
2026-05-21T02:22:04.123288346Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-21T02:22:04.236046823Z INFO: Gathering OVN_Northbound from ovnkube-node-5ttwc...
2026-05-21T02:22:04.236350945Z INFO: Gathering OVN_Northbound from ovnkube-node-p285b...
2026-05-21T02:22:04.236656968Z INFO: Gathering OVN_Northbound from ovnkube-node-z8sf4...
2026-05-21T02:22:04.236965162Z INFO: Gathering OVN_Southbound from ovnkube-node-5ttwc...
2026-05-21T02:22:04.237310738Z INFO: Gathering OVN_Southbound from ovnkube-node-p285b...
2026-05-21T02:22:04.237582718Z INFO: Gathering OVN_Southbound from ovnkube-node-z8sf4...
2026-05-21T02:22:04.385965676Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-21T02:22:04.436356647Z Wrote inspect data to must-gather.
2026-05-21T02:22:04.519605160Z tar: Removing leading `/' from member names
2026-05-21T02:22:04.549797515Z tar: Removing leading `/' from member names
2026-05-21T02:22:04.613007741Z tar: Removing leading `/' from member names
2026-05-21T02:22:04.613075148Z tar: Removing leading `/' from member names
2026-05-21T02:22:04.613359030Z tar: Removing leading `/' from member names
2026-05-21T02:22:04.687795769Z tar: Removing leading `/' from member names
2026-05-21T02:22:04.856056805Z Wrote inspect data to must-gather.
2026-05-21T02:22:04.893130338Z INFO: Getting status from alertmanager-main-0
2026-05-21T02:22:05.255797610Z Wrote inspect data to must-gather.
2026-05-21T02:22:05.321392282Z INFO: Waiting for network log collection to complete ...
2026-05-21T02:22:05.321513266Z INFO: Waiting for ovnk database copies to complete ...
2026-05-21T02:22:05.322077141Z INFO: Copying ovnk databases complete.
2026-05-21T02:22:05.323858216Z 4.3M	must-gather/network_logs/ovnk_database_store
2026-05-21T02:22:05.326125241Z ovnk_database_store/
2026-05-21T02:22:05.326176573Z ovnk_database_store/ovnkube-node-5ttwc_nbdb
2026-05-21T02:22:05.336626545Z ovnk_database_store/ovnkube-node-5ttwc_sbdb
2026-05-21T02:22:05.361922827Z ovnk_database_store/ovnkube-node-z8sf4_nbdb
2026-05-21T02:22:05.376283527Z ovnk_database_store/ovnkube-node-p285b_nbdb
2026-05-21T02:22:05.389637543Z ovnk_database_store/ovnkube-node-z8sf4_sbdb
2026-05-21T02:22:05.418760231Z ovnk_database_store/ovnkube-node-p285b_sbdb
2026-05-21T02:22:05.447888724Z INFO: Network log collection complete.
2026-05-21T02:22:05.582230724Z Wrote inspect data to must-gather.
2026-05-21T02:22:05.931940179Z W0521 02:22:05.931894    1413 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-21T02:22:05.947355260Z Wrote inspect data to must-gather.
2026-05-21T02:22:06.218824418Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-21T02:22:06.222950207Z INFO: Fallback to identify the container image from release info
2026-05-21T02:22:06.252629993Z Gathering data for ns/openshift-network-console...
2026-05-21T02:22:06.742590222Z Gathering data for ns/openshift-console-operator...
2026-05-21T02:22:06.956597941Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-21T02:22:07.083663194Z daemonset.apps/perf-node-gather-daemonset created
2026-05-21T02:22:07.177304284Z Gathering data for ns/openshift-console...
2026-05-21T02:22:07.193687027Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T02:22:07.229026491Z Waiting for performance profile collector pods to become ready: 1
2026-05-21T02:22:07.672292868Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-21T02:22:08.089868288Z Gathering data for ns/openshift-dns-operator...
2026-05-21T02:22:08.384025096Z Waiting for performance profile collector pods to become ready: 2
2026-05-21T02:22:08.420440282Z Gathering data for ns/openshift-dns...
2026-05-21T02:22:08.823518968Z Wrote inspect data to must-gather.
2026-05-21T02:22:08.981157270Z Gathering data for ns/openshift-image-registry...
2026-05-21T02:22:09.445548561Z Gathering data for ns/openshift-ingress-operator...
2026-05-21T02:22:09.522281550Z Waiting for performance profile collector pods to become ready: 3
2026-05-21T02:22:09.789726827Z Gathering data for ns/openshift-ingress...
2026-05-21T02:22:10.507676725Z Gathering data for ns/openshift-ingress-canary...
2026-05-21T02:22:10.634382512Z Waiting for performance profile collector pods to become ready: 4
2026-05-21T02:22:10.970427542Z Gathering data for ns/openshift-insights...
2026-05-21T02:22:11.752165879Z Waiting for performance profile collector pods to become ready: 5
2026-05-21T02:22:12.198234588Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T02:22:12.793702745Z Gathering data for ns/opendatahub...
2026-05-21T02:22:12.868161240Z Waiting for performance profile collector pods to become ready: 6
2026-05-21T02:22:13.979715051Z Waiting for performance profile collector pods to become ready: 7
2026-05-21T02:22:14.124305808Z Gathering data for ns/openshift-lws-operator...
2026-05-21T02:22:15.083365909Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-21T02:22:15.310980622Z Collecting performance related data for node ip-10-0-141-162.ec2.internal
2026-05-21T02:22:15.317533382Z Collecting performance related data for node ip-10-0-136-250.ec2.internal
2026-05-21T02:22:15.325093297Z Collecting performance related data for node ip-10-0-129-167.ec2.internal
2026-05-21T02:22:15.399835115Z Gathering data for ns/openshift-config...
2026-05-21T02:22:15.746516875Z Gathering data for ns/openshift-config-managed...
2026-05-21T02:22:16.317504167Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-21T02:22:16.779923265Z Gathering data for ns/openshift-kube-apiserver...
2026-05-21T02:22:17.137064593Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-21T02:22:17.205436956Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T02:22:17.445578484Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-21T02:22:17.810510841Z Gathering data for ns/openshift-kube-scheduler...
2026-05-21T02:22:18.139212042Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-21T02:22:18.489493046Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-21T02:22:18.783530358Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-21T02:22:19.115028410Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-21T02:22:19.643529420Z Gathering data for ns/openshift-multus...
2026-05-21T02:22:20.905514489Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-21T02:22:22.211844108Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T02:22:22.298363661Z Gathering data for ns/openshift-host-network...
2026-05-21T02:22:22.618927511Z Gathering data for ns/openshift-network-diagnostics...
2026-05-21T02:22:23.169530671Z Gathering data for ns/openshift-network-node-identity...
2026-05-21T02:22:23.600450312Z Collecting kubelet logs for node ip-10-0-141-162.ec2.internal
2026-05-21T02:22:23.602742579Z Collecting kubelet logs for node ip-10-0-136-250.ec2.internal
2026-05-21T02:22:23.604753264Z Collecting kubelet logs for node ip-10-0-129-167.ec2.internal
2026-05-21T02:22:23.605795445Z Gathering data for ns/openshift-network-operator...
2026-05-21T02:22:23.981127982Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-21T02:22:23.986897585Z INFO: Node performance data collection complete.
2026-05-21T02:22:24.031228495Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-21T02:22:24.325910019Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-21T02:22:24.755320676Z Gathering data for ns/openshift-apiserver-operator...
2026-05-21T02:22:25.065897467Z Gathering data for ns/openshift-apiserver...
2026-05-21T02:22:25.442659905Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-21T02:22:25.717678801Z Gathering data for ns/openshift-controller-manager...
2026-05-21T02:22:26.008279237Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-21T02:22:26.623879496Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-21T02:22:26.940208254Z Gathering data for ns/openshift-service-ca-operator...
2026-05-21T02:22:27.218692764Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T02:22:27.239730875Z Gathering data for ns/openshift-service-ca...
2026-05-21T02:22:27.612799406Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-21T02:22:28.522616565Z Wrote inspect data to must-gather.
2026-05-21T02:22:28.522662797Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T02:22:28.522662797Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-21T02:22:28.652892232Z error: the server doesn't have a resource type "clusters"
2026-05-21T02:22:28.831193730Z Caches written to disk