2026-05-06T17:46:56.322060935Z [disk usage checker] Started
2026-05-06T17:46:56.325427579Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:46:56.557919984Z Gathering data for ns/openshift-cluster-version...
2026-05-06T17:46:56.727028301Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-06T17:46:56.849319456Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-06T17:46:56.957445362Z Gathering data for ns/default...
2026-05-06T17:46:57.385476003Z Gathering data for ns/openshift...
2026-05-06T17:46:57.878597357Z Gathering data for ns/kube-system...
2026-05-06T17:46:58.654445362Z Gathering data for ns/openshift-etcd...
2026-05-06T17:46:58.825639712Z Waiting on subprocesses to finish execution.
2026-05-06T17:46:58.830626508Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-06T17:46:58.832164068Z INFO: Collecting host service logs for crio
2026-05-06T17:46:58.832444090Z INFO: Collecting host service logs for kubelet
2026-05-06T17:46:58.832744644Z INFO: Collecting host service logs for rpm-ostreed
2026-05-06T17:46:58.833043355Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-06T17:46:58.833360469Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-06T17:46:58.833653614Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-06T17:46:58.833950237Z INFO: Collecting host service logs for NetworkManager
2026-05-06T17:46:58.834266954Z INFO: Collecting host service logs for openvswitch
2026-05-06T17:46:58.834601915Z INFO: Gathering HAProxy config files
2026-05-06T17:46:58.834601915Z INFO: Collecting host service logs for ovs-configuration
2026-05-06T17:46:58.835000449Z INFO: Collecting host service logs for ovsdb-server
2026-05-06T17:46:58.835313944Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-06T17:46:58.835679491Z INFO: Waiting for worker host service log collection to complete ...
2026-05-06T17:46:58.837879028Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-06T17:46:58.848824020Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-06T17:46:58.936049920Z INFO: Waiting for node performance related collection to complete ...
2026-05-06T17:46:59.665211131Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-06T17:46:59.893887876Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-06T17:46:59.924318199Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-06T17:46:59.924403265Z INFO: on-disk MachineConfig config collection complete.
2026-05-06T17:47:00.020217352Z Wrote inspect data to must-gather.
2026-05-06T17:47:00.020333433Z error: inspection completed with the errors occurred while gathering data:
2026-05-06T17:47:00.020333433Z     namespaces "assisted-installer" not found
2026-05-06T17:47:00.037031080Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-06T17:47:00.123914738Z INFO: Collecting Insights Archives from insights-operator-544c98cc96-44vjf
2026-05-06T17:47:00.123973245Z insights-runtime-extractor-24dsh
2026-05-06T17:47:00.123995134Z insights-runtime-extractor-866jg
2026-05-06T17:47:00.124015817Z insights-runtime-extractor-tgn76
2026-05-06T17:47:00.128267714Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-06T17:47:00.137884917Z INFO: "metallb-operator" not detected. Skipping.
2026-05-06T17:47:00.149041992Z No resources found in openshift-etcd namespace.
2026-05-06T17:47:00.167635449Z INFO: Worker host service log collection to complete.
2026-05-06T17:47:00.170021693Z error: the server doesn't have a resource type "performanceprofile"
2026-05-06T17:47:00.305397119Z No resources found
2026-05-06T17:47:00.466385828Z Wrote inspect data to must-gather.
2026-05-06T17:47:00.482308579Z ERROR: No running kube-apiserver pods found
2026-05-06T17:47:00.710122548Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-06T17:47:00.710122548Z See 'oc rsync -h' for help and examples
2026-05-06T17:47:00.762812133Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-06T17:47:01.118872917Z Wrote inspect data to must-gather.
2026-05-06T17:47:01.180329243Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-06T17:47:01.186777278Z No resources found
2026-05-06T17:47:01.192116723Z INFO: 'previous-logs' folder not found on ip-10-0-130-186.ec2.internal, skipping...
2026-05-06T17:47:01.244718071Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-06T17:47:01.292847140Z Wrote inspect data to must-gather.
2026-05-06T17:47:01.375987996Z Wrote inspect data to must-gather.
2026-05-06T17:47:01.400705892Z Wrote inspect data to must-gather.
2026-05-06T17:47:01.454563739Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:47:01.490629137Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-06T17:47:01.662041379Z error: the server doesn't have a resource type "machineconfigs"
2026-05-06T17:47:01.730393336Z INFO: 'previous-logs' folder not found on ip-10-0-131-115.ec2.internal, skipping...
2026-05-06T17:47:01.768561733Z tar: Removing leading `/' from member names
2026-05-06T17:47:01.769307137Z Wrote inspect data to must-gather.
2026-05-06T17:47:01.774631582Z INFO: HAProxy config collection complete.
2026-05-06T17:47:01.897545428Z Gathering data for ns/kuadrant-system...
2026-05-06T17:47:01.991243570Z Wrote inspect data to must-gather.
2026-05-06T17:47:02.030665561Z INFO: Getting rules from prometheus-k8s-0
2026-05-06T17:47:02.048888359Z INFO: 'previous-logs' folder not found on ip-10-0-135-110.ec2.internal, skipping...
2026-05-06T17:47:02.048941341Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-06T17:47:02.048941341Z INFO: Machine Config Daemon termination log collection complete.
2026-05-06T17:47:02.149710418Z Wrote inspect data to must-gather.
2026-05-06T17:47:02.380697769Z Wrote inspect data to must-gather.
2026-05-06T17:47:02.565133907Z INFO: Getting status/config from prometheus-k8s-0
2026-05-06T17:47:02.567068741Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-06T17:47:02.684444069Z Wrote inspect data to must-gather.
2026-05-06T17:47:02.766555540Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-06T17:47:02.931198535Z Wrote inspect data to must-gather.
2026-05-06T17:47:03.066608402Z Wrote inspect data to must-gather.
2026-05-06T17:47:03.074972329Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-06T17:47:03.253632980Z W0506 17:47:03.253593    1145 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-06T17:47:03.294189961Z Wrote inspect data to must-gather.
2026-05-06T17:47:03.336016747Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-06T17:47:03.367559878Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-06T17:47:03.560857135Z INFO: INTERCONNECT MODE
2026-05-06T17:47:03.561381675Z INFO: Gathering ovn-kubernetes DBs
2026-05-06T17:47:03.592073524Z Gathering data for ns/openshift-monitoring...
2026-05-06T17:47:03.650222705Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-06T17:47:03.667307385Z Wrote inspect data to must-gather.
2026-05-06T17:47:03.762064157Z INFO: Gathering OVN_Northbound from ovnkube-node-9s6rp...
2026-05-06T17:47:03.763381476Z INFO: Gathering OVN_Northbound from ovnkube-node-bbxnx...
2026-05-06T17:47:03.763650048Z INFO: Gathering OVN_Northbound from ovnkube-node-lz4g2...
2026-05-06T17:47:03.763980873Z INFO: Gathering OVN_Southbound from ovnkube-node-9s6rp...
2026-05-06T17:47:03.764306133Z INFO: Gathering OVN_Southbound from ovnkube-node-bbxnx...
2026-05-06T17:47:03.764615125Z INFO: Gathering OVN_Southbound from ovnkube-node-lz4g2...
2026-05-06T17:47:04.004330096Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-06T17:47:04.129693651Z tar: Removing leading `/' from member names
2026-05-06T17:47:04.139089111Z tar: Removing leading `/' from member names
2026-05-06T17:47:04.158087074Z tar: Removing leading `/' from member names
2026-05-06T17:47:04.189630002Z tar: Removing leading `/' from member names
2026-05-06T17:47:04.202733559Z tar: Removing leading `/' from member names
2026-05-06T17:47:04.213372625Z tar: Removing leading `/' from member names
2026-05-06T17:47:04.217647692Z Wrote inspect data to must-gather.
2026-05-06T17:47:04.597694250Z INFO: Getting status from alertmanager-main-0
2026-05-06T17:47:04.607546479Z Wrote inspect data to must-gather.
2026-05-06T17:47:04.803909315Z INFO: Waiting for network log collection to complete ...
2026-05-06T17:47:04.803968142Z INFO: Waiting for ovnk database copies to complete ...
2026-05-06T17:47:04.804473804Z INFO: Copying ovnk databases complete.
2026-05-06T17:47:04.806259827Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-05-06T17:47:04.809106980Z ovnk_database_store/
2026-05-06T17:47:04.809206008Z ovnk_database_store/ovnkube-node-9s6rp_nbdb
2026-05-06T17:47:04.819468915Z ovnk_database_store/ovnkube-node-bbxnx_nbdb
2026-05-06T17:47:04.829750296Z ovnk_database_store/ovnkube-node-lz4g2_nbdb
2026-05-06T17:47:04.839018501Z ovnk_database_store/ovnkube-node-lz4g2_sbdb
2026-05-06T17:47:04.862328999Z ovnk_database_store/ovnkube-node-bbxnx_sbdb
2026-05-06T17:47:04.889357117Z ovnk_database_store/ovnkube-node-9s6rp_sbdb
2026-05-06T17:47:04.922936136Z INFO: Network log collection complete.
2026-05-06T17:47:04.961277578Z Wrote inspect data to must-gather.
2026-05-06T17:47:05.268668636Z Wrote inspect data to must-gather.
2026-05-06T17:47:05.544245747Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-06T17:47:05.547494499Z INFO: Fallback to identify the container image from release info
2026-05-06T17:47:05.929614766Z Gathering data for ns/openshift-network-console...
2026-05-06T17:47:06.152614847Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e33dd3daea88bad2786194ceeeebe5712cd3befc2bbc8ace6a463156f9f7e36f
2026-05-06T17:47:06.289739208Z daemonset.apps/perf-node-gather-daemonset created
2026-05-06T17:47:06.457261000Z Waiting for performance profile collector pods to become ready: 1
2026-05-06T17:47:06.477411363Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:47:06.511449926Z Gathering data for ns/openshift-console-operator...
2026-05-06T17:47:07.000694075Z Gathering data for ns/openshift-console...
2026-05-06T17:47:07.550166535Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-06T17:47:07.613080965Z Waiting for performance profile collector pods to become ready: 2
2026-05-06T17:47:08.042254015Z Gathering data for ns/openshift-dns-operator...
2026-05-06T17:47:08.259437641Z Wrote inspect data to must-gather.
2026-05-06T17:47:08.432774974Z Gathering data for ns/openshift-dns...
2026-05-06T17:47:08.720237544Z Waiting for performance profile collector pods to become ready: 3
2026-05-06T17:47:09.218224919Z Gathering data for ns/openshift-image-registry...
2026-05-06T17:47:09.768282690Z Gathering data for ns/openshift-ingress-operator...
2026-05-06T17:47:09.858642355Z Waiting for performance profile collector pods to become ready: 4
2026-05-06T17:47:10.235775089Z Gathering data for ns/openshift-ingress...
2026-05-06T17:47:10.929444150Z Gathering data for ns/openshift-ingress-canary...
2026-05-06T17:47:10.977333643Z Waiting for performance profile collector pods to become ready: 5
2026-05-06T17:47:11.405012186Z Gathering data for ns/openshift-insights...
2026-05-06T17:47:11.482051091Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:47:12.109976394Z Waiting for performance profile collector pods to become ready: 6
2026-05-06T17:47:13.225107734Z Waiting for performance profile collector pods to become ready: 7
2026-05-06T17:47:13.515460088Z Gathering data for ns/opendatahub...
2026-05-06T17:47:14.354585616Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-06T17:47:14.584191333Z Collecting performance related data for node ip-10-0-135-110.ec2.internal
2026-05-06T17:47:14.590719689Z Collecting performance related data for node ip-10-0-131-115.ec2.internal
2026-05-06T17:47:14.598383395Z Collecting performance related data for node ip-10-0-130-186.ec2.internal
2026-05-06T17:47:14.974626654Z Gathering data for ns/openshift-lws-operator...
2026-05-06T17:47:16.458356938Z Gathering data for ns/openshift-config...
2026-05-06T17:47:16.488667262Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:47:16.824765658Z Gathering data for ns/openshift-config-managed...
2026-05-06T17:47:17.425176885Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-06T17:47:17.782789683Z Gathering data for ns/openshift-kube-apiserver...
2026-05-06T17:47:18.150611295Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-06T17:47:18.506427935Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-06T17:47:18.855204826Z Gathering data for ns/openshift-kube-scheduler...
2026-05-06T17:47:19.194112960Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-06T17:47:19.573737042Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-06T17:47:19.987278500Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-06T17:47:20.370124044Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-06T17:47:20.994320018Z Gathering data for ns/openshift-multus...
2026-05-06T17:47:21.493223152Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:47:22.731270223Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-06T17:47:22.980619274Z Collecting kubelet logs for node ip-10-0-135-110.ec2.internal
2026-05-06T17:47:22.983371772Z Collecting kubelet logs for node ip-10-0-131-115.ec2.internal
2026-05-06T17:47:22.986079677Z Collecting kubelet logs for node ip-10-0-130-186.ec2.internal
2026-05-06T17:47:23.349378334Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-06T17:47:23.354095036Z INFO: Node performance data collection complete.
2026-05-06T17:47:24.293985239Z Gathering data for ns/openshift-host-network...
2026-05-06T17:47:24.664429190Z Gathering data for ns/openshift-network-diagnostics...
2026-05-06T17:47:25.255520951Z Gathering data for ns/openshift-network-node-identity...
2026-05-06T17:47:25.743423494Z Gathering data for ns/openshift-network-operator...
2026-05-06T17:47:26.181347468Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-06T17:47:26.499500814Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:47:26.525832133Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-06T17:47:27.018075788Z Gathering data for ns/openshift-apiserver-operator...
2026-05-06T17:47:27.407582804Z Gathering data for ns/openshift-apiserver...
2026-05-06T17:47:27.883433076Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-06T17:47:28.213127441Z Gathering data for ns/openshift-controller-manager...
2026-05-06T17:47:28.567551496Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-06T17:47:29.280911814Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-06T17:47:29.618293440Z Gathering data for ns/openshift-service-ca-operator...
2026-05-06T17:47:29.994089213Z Gathering data for ns/openshift-service-ca...
2026-05-06T17:47:30.411189915Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-06T17:47:31.423263316Z Wrote inspect data to must-gather.
2026-05-06T17:47:31.423307969Z error: inspection completed with the errors occurred while gathering data:
2026-05-06T17:47:31.423307969Z     [skipping gathering secrets/support due to error: secrets "support" not found, skipping gathering customresourcedefinitions.apiextensions.k8s.io due to error: skipping gathering namespaces/opendatahub due to error: one or more errors occurred while gathering pod-specific data for namespace: opendatahub
2026-05-06T17:47:31.423307969Z 
2026-05-06T17:47:31.423307969Z     [one or more errors occurred while gathering container data for pod maas-api-5c87f7f96f-dmssh:
2026-05-06T17:47:31.423307969Z 
2026-05-06T17:47:31.423307969Z     [container "maas-api" in pod "maas-api-5c87f7f96f-dmssh" is waiting to start: CreateContainerConfigError, previous terminated container "maas-api" in pod "maas-api-5c87f7f96f-dmssh" not found, the server could not find the requested resource], one or more errors occurred while gathering container data for pod maas-api-7f4595bb66-9kzqf:
2026-05-06T17:47:31.423307969Z 
2026-05-06T17:47:31.423307969Z     [previous terminated container "maas-api" in pod "maas-api-7f4595bb66-9kzqf" not found, container "maas-api" in pod "maas-api-7f4595bb66-9kzqf" is waiting to start: CreateContainerConfigError, the server could not find the requested resource]]]
2026-05-06T17:47:31.504246359Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-05-06T17:47:31.555642624Z error: the server doesn't have a resource type "clusters"
2026-05-06T17:47:31.731537763Z Caches written to disk