2026-05-21T19:07:57.120003511Z [disk usage checker] Started
2026-05-21T19:07:57.123389145Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T19:07:57.348091864Z Gathering data for ns/openshift-cluster-version...
2026-05-21T19:07:57.541934551Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T19:07:57.668585990Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T19:07:57.782869507Z Gathering data for ns/default...
2026-05-21T19:07:58.166724341Z Gathering data for ns/openshift...
2026-05-21T19:07:58.686410222Z Gathering data for ns/kube-system...
2026-05-21T19:07:59.522497224Z Waiting on subprocesses to finish execution.
2026-05-21T19:07:59.528372613Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-21T19:07:59.530723770Z INFO: Gathering HAProxy config files
2026-05-21T19:07:59.544593538Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-21T19:07:59.545288345Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-21T19:07:59.546023457Z INFO: Collecting host service logs for crio
2026-05-21T19:07:59.546319378Z INFO: Collecting host service logs for kubelet
2026-05-21T19:07:59.546637626Z INFO: Collecting host service logs for rpm-ostreed
2026-05-21T19:07:59.546985903Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-21T19:07:59.547304605Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-21T19:07:59.547663971Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-21T19:07:59.548014694Z INFO: Collecting host service logs for NetworkManager
2026-05-21T19:07:59.548317731Z INFO: Collecting host service logs for openvswitch
2026-05-21T19:07:59.548627768Z INFO: Collecting host service logs for ovs-configuration
2026-05-21T19:07:59.548969039Z INFO: Collecting host service logs for ovsdb-server
2026-05-21T19:07:59.549277883Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-21T19:07:59.549646458Z INFO: Waiting for worker host service log collection to complete ...
2026-05-21T19:07:59.641369529Z Gathering data for ns/openshift-etcd...
2026-05-21T19:07:59.690040473Z INFO: Waiting for node performance related collection to complete ...
2026-05-21T19:08:00.023304335Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-21T19:08:00.023398332Z INFO: on-disk MachineConfig config collection complete.
2026-05-21T19:08:00.483303455Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-21T19:08:00.592108806Z No resources found
2026-05-21T19:08:00.606333899Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-jvb29
2026-05-21T19:08:00.607026934Z insights-runtime-extractor-7zs94
2026-05-21T19:08:00.607099116Z insights-runtime-extractor-f92gd
2026-05-21T19:08:00.607124429Z insights-runtime-extractor-x5p2z
2026-05-21T19:08:00.687198350Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-21T19:08:00.734510551Z Wrote inspect data to must-gather.
2026-05-21T19:08:00.794248862Z error: the server doesn't have a resource type "performanceprofile"
2026-05-21T19:08:00.880142735Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-21T19:08:00.940063205Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-21T19:08:01.041064058Z INFO: "metallb-operator" not detected. Skipping.
2026-05-21T19:08:01.171276869Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-21T19:08:01.392367777Z No resources found in openshift-etcd namespace.
2026-05-21T19:08:01.428497511Z INFO: Worker host service log collection to complete.
2026-05-21T19:08:01.607442479Z Wrote inspect data to must-gather.
2026-05-21T19:08:01.611819360Z Wrote inspect data to must-gather.
2026-05-21T19:08:01.611912315Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T19:08:01.611912315Z     namespaces "assisted-installer" not found
2026-05-21T19:08:01.733705748Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-21T19:08:01.733705748Z See 'oc rsync -h' for help and examples
2026-05-21T19:08:01.780772636Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-21T19:08:01.837973225Z ERROR: No running kube-apiserver pods found
2026-05-21T19:08:01.877987629Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-21T19:08:01.977333287Z No resources found
2026-05-21T19:08:02.165628336Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T19:08:02.226564895Z Wrote inspect data to must-gather.
2026-05-21T19:08:02.266177661Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T19:08:02.289832906Z Wrote inspect data to must-gather.
2026-05-21T19:08:02.298773395Z Gathering data for ns/kuadrant-system...
2026-05-21T19:08:02.337146512Z Wrote inspect data to must-gather.
2026-05-21T19:08:02.370411541Z Wrote inspect data to must-gather.
2026-05-21T19:08:02.387608984Z INFO: 'previous-logs' folder not found on ip-10-0-128-227.ec2.internal, skipping...
2026-05-21T19:08:02.493125109Z tar: Removing leading `/' from member names
2026-05-21T19:08:02.501654063Z INFO: HAProxy config collection complete.
2026-05-21T19:08:02.585055560Z error: the server doesn't have a resource type "machineconfigs"
2026-05-21T19:08:02.692007472Z INFO: Getting rules from prometheus-k8s-0
2026-05-21T19:08:02.693029099Z Wrote inspect data to must-gather.
2026-05-21T19:08:02.720621653Z INFO: 'previous-logs' folder not found on ip-10-0-134-192.ec2.internal, skipping...
2026-05-21T19:08:02.920790966Z Wrote inspect data to must-gather.
2026-05-21T19:08:03.056586404Z INFO: 'previous-logs' folder not found on ip-10-0-135-21.ec2.internal, skipping...
2026-05-21T19:08:03.056623765Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-21T19:08:03.056651618Z INFO: Machine Config Daemon termination log collection complete.
2026-05-21T19:08:03.078691739Z Wrote inspect data to must-gather.
2026-05-21T19:08:03.189823712Z INFO: Getting status/config from prometheus-k8s-0
2026-05-21T19:08:03.235897304Z Wrote inspect data to must-gather.
2026-05-21T19:08:03.453304902Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-21T19:08:03.453933270Z Wrote inspect data to must-gather.
2026-05-21T19:08:03.459138436Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T19:08:03.742234988Z Wrote inspect data to must-gather.
2026-05-21T19:08:03.757197307Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-21T19:08:03.803694241Z INFO: INTERCONNECT MODE
2026-05-21T19:08:03.803739826Z INFO: Gathering ovn-kubernetes DBs
2026-05-21T19:08:04.005767549Z W0521 19:08:04.005704    1163 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-21T19:08:04.040023750Z Wrote inspect data to must-gather.
2026-05-21T19:08:04.045646627Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T19:08:04.048440404Z INFO: Gathering OVN_Northbound from ovnkube-node-8przh...
2026-05-21T19:08:04.048679701Z INFO: Gathering OVN_Northbound from ovnkube-node-cpdw4...
2026-05-21T19:08:04.048963112Z INFO: Gathering OVN_Northbound from ovnkube-node-vp6dm...
2026-05-21T19:08:04.049258558Z INFO: Gathering OVN_Southbound from ovnkube-node-8przh...
2026-05-21T19:08:04.049523915Z INFO: Gathering OVN_Southbound from ovnkube-node-cpdw4...
2026-05-21T19:08:04.049798920Z INFO: Gathering OVN_Southbound from ovnkube-node-vp6dm...
2026-05-21T19:08:04.066390408Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-21T19:08:04.376446148Z Gathering data for ns/openshift-monitoring...
2026-05-21T19:08:04.385670408Z tar: Removing leading `/' from member names
2026-05-21T19:08:04.441819305Z tar: Removing leading `/' from member names
2026-05-21T19:08:04.510916465Z tar: Removing leading `/' from member names
2026-05-21T19:08:04.515553037Z tar: Removing leading `/' from member names
2026-05-21T19:08:04.519915135Z tar: Removing leading `/' from member names
2026-05-21T19:08:04.602791022Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-21T19:08:04.639886131Z Wrote inspect data to must-gather.
2026-05-21T19:08:04.883597930Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-21T19:08:05.041111284Z Wrote inspect data to must-gather.
2026-05-21T19:08:05.202056932Z INFO: Waiting for network log collection to complete ...
2026-05-21T19:08:05.202096434Z INFO: Waiting for ovnk database copies to complete ...
2026-05-21T19:08:05.353288684Z INFO: Getting status from alertmanager-main-0
2026-05-21T19:08:05.359662564Z Wrote inspect data to must-gather.
2026-05-21T19:08:05.473957883Z tar: Removing leading `/' from member names
2026-05-21T19:08:05.489870390Z INFO: Copying ovnk databases complete.
2026-05-21T19:08:05.492261218Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-05-21T19:08:05.495986105Z ovnk_database_store/
2026-05-21T19:08:05.496029896Z ovnk_database_store/ovnkube-node-cpdw4_nbdb
2026-05-21T19:08:05.512357795Z ovnk_database_store/ovnkube-node-vp6dm_sbdb
2026-05-21T19:08:05.536132509Z ovnk_database_store/ovnkube-node-8przh_nbdb
2026-05-21T19:08:05.548379124Z ovnk_database_store/ovnkube-node-vp6dm_nbdb
2026-05-21T19:08:05.560587884Z ovnk_database_store/ovnkube-node-cpdw4_sbdb
2026-05-21T19:08:05.589772726Z ovnk_database_store/ovnkube-node-8przh_sbdb
2026-05-21T19:08:05.614943650Z INFO: Network log collection complete.
2026-05-21T19:08:05.661243494Z Wrote inspect data to must-gather.
2026-05-21T19:08:05.976371506Z Wrote inspect data to must-gather.
2026-05-21T19:08:06.264323440Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-21T19:08:06.267306917Z INFO: Fallback to identify the container image from release info
2026-05-21T19:08:06.735432756Z Gathering data for ns/openshift-network-console...
2026-05-21T19:08:06.984955152Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-21T19:08:07.123079605Z daemonset.apps/perf-node-gather-daemonset created
2026-05-21T19:08:07.202501189Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T19:08:07.287258892Z Gathering data for ns/openshift-console-operator...
2026-05-21T19:08:07.288591347Z Waiting for performance profile collector pods to become ready: 1
2026-05-21T19:08:07.880055299Z Gathering data for ns/openshift-console...
2026-05-21T19:08:08.420005535Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-21T19:08:08.436346698Z Waiting for performance profile collector pods to become ready: 2
2026-05-21T19:08:08.868396400Z Gathering data for ns/openshift-dns-operator...
2026-05-21T19:08:09.153434417Z Wrote inspect data to must-gather.
2026-05-21T19:08:09.288167778Z Gathering data for ns/openshift-dns...
2026-05-21T19:08:09.544739094Z Waiting for performance profile collector pods to become ready: 3
2026-05-21T19:08:09.946424961Z Gathering data for ns/openshift-image-registry...
2026-05-21T19:08:10.447963333Z Gathering data for ns/openshift-ingress-operator...
2026-05-21T19:08:10.651733898Z Waiting for performance profile collector pods to become ready: 4
2026-05-21T19:08:10.894085065Z Gathering data for ns/openshift-ingress...
2026-05-21T19:08:11.716800742Z Gathering data for ns/openshift-ingress-canary...
2026-05-21T19:08:11.771518409Z Waiting for performance profile collector pods to become ready: 5
2026-05-21T19:08:12.197410103Z Gathering data for ns/openshift-insights...
2026-05-21T19:08:12.209310070Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T19:08:12.898062522Z Waiting for performance profile collector pods to become ready: 6
2026-05-21T19:08:14.014538623Z Waiting for performance profile collector pods to become ready: 7
2026-05-21T19:08:14.309140871Z Gathering data for ns/opendatahub...
2026-05-21T19:08:15.134954687Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-21T19:08:15.379718012Z Collecting performance related data for node ip-10-0-135-21.ec2.internal
2026-05-21T19:08:15.387478322Z Collecting performance related data for node ip-10-0-134-192.ec2.internal
2026-05-21T19:08:15.402800422Z Collecting performance related data for node ip-10-0-128-227.ec2.internal
2026-05-21T19:08:15.859889056Z Gathering data for ns/openshift-lws-operator...
2026-05-21T19:08:17.213621365Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T19:08:17.281231056Z Gathering data for ns/openshift-config...
2026-05-21T19:08:17.620130384Z Gathering data for ns/openshift-config-managed...
2026-05-21T19:08:18.171117686Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-21T19:08:18.522105540Z Gathering data for ns/openshift-kube-apiserver...
2026-05-21T19:08:18.903742916Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-21T19:08:19.252813851Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-21T19:08:19.631967170Z Gathering data for ns/openshift-kube-scheduler...
2026-05-21T19:08:19.968605565Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-21T19:08:20.332947596Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-21T19:08:20.770173482Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-21T19:08:21.138965239Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-21T19:08:21.731651581Z Gathering data for ns/openshift-multus...
2026-05-21T19:08:22.217940606Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T19:08:23.145527427Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-21T19:08:23.849472954Z Collecting kubelet logs for node ip-10-0-135-21.ec2.internal
2026-05-21T19:08:23.851552921Z Collecting kubelet logs for node ip-10-0-134-192.ec2.internal
2026-05-21T19:08:23.853586599Z Collecting kubelet logs for node ip-10-0-128-227.ec2.internal
2026-05-21T19:08:24.221642597Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-21T19:08:24.226727526Z INFO: Node performance data collection complete.
2026-05-21T19:08:24.700223143Z Gathering data for ns/openshift-host-network...
2026-05-21T19:08:25.057437739Z Gathering data for ns/openshift-network-diagnostics...
2026-05-21T19:08:25.618004226Z Gathering data for ns/openshift-network-node-identity...
2026-05-21T19:08:26.064660844Z Gathering data for ns/openshift-network-operator...
2026-05-21T19:08:26.502986623Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-21T19:08:26.837714218Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-21T19:08:27.224667050Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T19:08:27.375937026Z Gathering data for ns/openshift-apiserver-operator...
2026-05-21T19:08:27.736446811Z Gathering data for ns/openshift-apiserver...
2026-05-21T19:08:28.179609910Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-21T19:08:28.504995288Z Gathering data for ns/openshift-controller-manager...
2026-05-21T19:08:28.850874802Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-21T19:08:29.525521103Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-21T19:08:29.891465101Z Gathering data for ns/openshift-service-ca-operator...
2026-05-21T19:08:30.230830288Z Gathering data for ns/openshift-service-ca...
2026-05-21T19:08:30.693801360Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-21T19:08:31.718906700Z Wrote inspect data to must-gather.
2026-05-21T19:08:31.718950094Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T19:08:31.718950094Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-21T19:08:31.852851070Z error: the server doesn't have a resource type "clusters"
2026-05-21T19:08:32.015512849Z Caches written to disk