2026-04-22T18:08:37.038129460Z [disk usage checker] Started
2026-04-22T18:08:37.041943747Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-04-22T18:08:37.433915558Z Gathering data for ns/openshift-cluster-version...
2026-04-22T18:08:37.599021126Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T18:08:37.694876548Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T18:08:37.796509332Z Gathering data for ns/default...
2026-04-22T18:08:38.137047050Z Gathering data for ns/openshift...
2026-04-22T18:08:38.590508805Z Gathering data for ns/kube-system...
2026-04-22T18:08:39.252908177Z Gathering data for ns/openshift-etcd...
2026-04-22T18:08:39.581995306Z Wrote inspect data to must-gather.
2026-04-22T18:08:39.582028907Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T18:08:39.582028907Z     namespaces "assisted-installer" not found
2026-04-22T18:08:39.870625655Z Waiting on subprocesses to finish execution.
2026-04-22T18:08:39.876985421Z INFO: Collecting host service logs for crio
2026-04-22T18:08:39.877551769Z INFO: Collecting host service logs for kubelet
2026-04-22T18:08:39.877934611Z INFO: Collecting host service logs for rpm-ostreed
2026-04-22T18:08:39.878275135Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-22T18:08:39.878624511Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-22T18:08:39.878989999Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-22T18:08:39.879274110Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-22T18:08:39.879374248Z INFO: Collecting host service logs for NetworkManager
2026-04-22T18:08:39.879744736Z INFO: Collecting host service logs for openvswitch
2026-04-22T18:08:39.880250195Z INFO: Collecting host service logs for ovs-configuration
2026-04-22T18:08:39.880619299Z INFO: Collecting host service logs for ovsdb-server
2026-04-22T18:08:39.880996718Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-22T18:08:39.881382787Z INFO: Waiting for worker host service log collection to complete ...
2026-04-22T18:08:39.887528398Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-22T18:08:39.904714789Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-22T18:08:39.909448230Z INFO: Gathering HAProxy config files
2026-04-22T18:08:39.954302163Z INFO: Waiting for node performance related collection to complete ...
2026-04-22T18:08:40.265400278Z No resources found
2026-04-22T18:08:40.459714019Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-22T18:08:40.723452958Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-k6554
2026-04-22T18:08:40.723510357Z insights-runtime-extractor-b68fk
2026-04-22T18:08:40.723532590Z insights-runtime-extractor-w688w
2026-04-22T18:08:40.723606841Z insights-runtime-extractor-xq75n
2026-04-22T18:08:41.111939396Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-22T18:08:41.374769513Z No resources found
2026-04-22T18:08:41.385371802Z error: the server doesn't have a resource type "performanceprofile"
2026-04-22T18:08:41.498934416Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T18:08:41.577830758Z Wrote inspect data to must-gather.
2026-04-22T18:08:41.779322268Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-22T18:08:41.779419017Z INFO: on-disk MachineConfig config collection complete.
2026-04-22T18:08:41.821654652Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-22T18:08:41.846228598Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-22T18:08:41.901395475Z INFO: Worker host service log collection to complete.
2026-04-22T18:08:41.935537593Z Wrote inspect data to must-gather.
2026-04-22T18:08:41.952426114Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-22T18:08:41.952426114Z See 'oc rsync -h' for help and examples
2026-04-22T18:08:41.962963014Z No resources found in openshift-etcd namespace.
2026-04-22T18:08:41.972128489Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-22T18:08:42.053057905Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-04-22T18:08:42.128281641Z ERROR: No running kube-apiserver pods found
2026-04-22T18:08:42.133067842Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-22T18:08:42.160093060Z Wrote inspect data to must-gather.
2026-04-22T18:08:42.183731020Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-22T18:08:42.238382832Z INFO: "metallb-operator" not detected. Skipping.
2026-04-22T18:08:42.460469343Z Wrote inspect data to must-gather.
2026-04-22T18:08:42.549770486Z Wrote inspect data to must-gather.
2026-04-22T18:08:42.557159555Z Wrote inspect data to must-gather.
2026-04-22T18:08:42.562043478Z Wrote inspect data to must-gather.
2026-04-22T18:08:42.610925551Z tar: Removing leading `/' from member names
2026-04-22T18:08:42.616844752Z INFO: HAProxy config collection complete.
2026-04-22T18:08:42.826245812Z INFO: 'previous-logs' folder not found on ip-10-0-133-169.ec2.internal, skipping...
2026-04-22T18:08:42.858864138Z error: the server doesn't have a resource type "machineconfigs"
2026-04-22T18:08:42.871449799Z INFO: Getting rules from prometheus-k8s-0
2026-04-22T18:08:43.002922551Z Wrote inspect data to must-gather.
2026-04-22T18:08:43.180216826Z Gathering data for ns/kuadrant-system...
2026-04-22T18:08:43.223109858Z Wrote inspect data to must-gather.
2026-04-22T18:08:43.248917968Z INFO: 'previous-logs' folder not found on ip-10-0-135-36.ec2.internal, skipping...
2026-04-22T18:08:43.401535934Z Wrote inspect data to must-gather.
2026-04-22T18:08:43.460399214Z INFO: Getting status/config from prometheus-k8s-0
2026-04-22T18:08:43.596070411Z Wrote inspect data to must-gather.
2026-04-22T18:08:43.726318534Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T18:08:43.748278496Z INFO: 'previous-logs' folder not found on ip-10-0-143-54.ec2.internal, skipping...
2026-04-22T18:08:43.748308356Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-22T18:08:43.748318651Z INFO: Machine Config Daemon termination log collection complete.
2026-04-22T18:08:43.816440608Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-22T18:08:43.852162455Z INFO: INTERCONNECT MODE
2026-04-22T18:08:43.852269061Z INFO: Gathering ovn-kubernetes DBs
2026-04-22T18:08:44.063122550Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T18:08:44.076347078Z INFO: Gathering OVN_Northbound from ovnkube-node-2b8mb...
2026-04-22T18:08:44.076671669Z INFO: Gathering OVN_Northbound from ovnkube-node-hrsxq...
2026-04-22T18:08:44.076973752Z INFO: Gathering OVN_Northbound from ovnkube-node-vckbs...
2026-04-22T18:08:44.077315922Z INFO: Gathering OVN_Southbound from ovnkube-node-2b8mb...
2026-04-22T18:08:44.077582624Z INFO: Gathering OVN_Southbound from ovnkube-node-hrsxq...
2026-04-22T18:08:44.077868057Z INFO: Gathering OVN_Southbound from ovnkube-node-vckbs...
2026-04-22T18:08:44.109011201Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-22T18:08:44.330313706Z Wrote inspect data to must-gather.
2026-04-22T18:08:44.347017597Z tar: Removing leading `/' from member names
2026-04-22T18:08:44.411145587Z tar: Removing leading `/' from member names
2026-04-22T18:08:44.439311481Z Gathering data for ns/openshift-monitoring...
2026-04-22T18:08:44.472734895Z tar: Removing leading `/' from member names
2026-04-22T18:08:44.481472938Z tar: Removing leading `/' from member names
2026-04-22T18:08:44.482442891Z tar: Removing leading `/' from member names
2026-04-22T18:08:44.528634906Z tar: Removing leading `/' from member names
2026-04-22T18:08:44.646510853Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-22T18:08:44.755913704Z Wrote inspect data to must-gather.
2026-04-22T18:08:44.913487658Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-22T18:08:45.110073583Z Wrote inspect data to must-gather.
2026-04-22T18:08:45.223890423Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-22T18:08:45.235334025Z INFO: Waiting for network log collection to complete ...
2026-04-22T18:08:45.235626130Z INFO: Waiting for ovnk database copies to complete ...
2026-04-22T18:08:45.236187095Z INFO: Copying ovnk databases complete.
2026-04-22T18:08:45.238218483Z 4.9M	must-gather/network_logs/ovnk_database_store
2026-04-22T18:08:45.241041695Z ovnk_database_store/
2026-04-22T18:08:45.241041695Z ovnk_database_store/ovnkube-node-vckbs_nbdb
2026-04-22T18:08:45.255174256Z ovnk_database_store/ovnkube-node-vckbs_sbdb
2026-04-22T18:08:45.287543444Z ovnk_database_store/ovnkube-node-2b8mb_nbdb
2026-04-22T18:08:45.300676225Z ovnk_database_store/ovnkube-node-hrsxq_nbdb
2026-04-22T18:08:45.315181721Z ovnk_database_store/ovnkube-node-hrsxq_sbdb
2026-04-22T18:08:45.352685152Z ovnk_database_store/ovnkube-node-2b8mb_sbdb
2026-04-22T18:08:45.389863076Z INFO: Network log collection complete.
2026-04-22T18:08:45.435343229Z Wrote inspect data to must-gather.
2026-04-22T18:08:45.660492466Z INFO: Getting status from alertmanager-main-0
2026-04-22T18:08:45.692104735Z Wrote inspect data to must-gather.
2026-04-22T18:08:45.984841772Z Wrote inspect data to must-gather.
2026-04-22T18:08:46.251797335Z Wrote inspect data to must-gather.
2026-04-22T18:08:46.553001665Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-22T18:08:46.557200661Z INFO: Fallback to identify the container image from release info
2026-04-22T18:08:46.838727515Z Gathering data for ns/openshift-network-console...
2026-04-22T18:08:47.060001463Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-04-22T18:08:47.218926485Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-22T18:08:47.334896829Z Gathering data for ns/openshift-console-operator...
2026-04-22T18:08:47.373407330Z daemonset.apps/perf-node-gather-daemonset created
2026-04-22T18:08:47.555405874Z Waiting for performance profile collector pods to become ready: 1
2026-04-22T18:08:47.809919200Z Gathering data for ns/openshift-console...
2026-04-22T18:08:48.014030984Z Wrote inspect data to must-gather.
2026-04-22T18:08:48.321803898Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-22T18:08:48.678639187Z Gathering data for ns/openshift-dns-operator...
2026-04-22T18:08:48.705732756Z Waiting for performance profile collector pods to become ready: 2
2026-04-22T18:08:48.975096665Z Gathering data for ns/openshift-dns...
2026-04-22T18:08:49.562522340Z Gathering data for ns/openshift-image-registry...
2026-04-22T18:08:49.846240529Z Waiting for performance profile collector pods to become ready: 3
2026-04-22T18:08:50.045845910Z Gathering data for ns/openshift-ingress-operator...
2026-04-22T18:08:50.378808069Z Gathering data for ns/openshift-ingress...
2026-04-22T18:08:50.899556930Z Gathering data for ns/openshift-ingress-canary...
2026-04-22T18:08:50.974850709Z Waiting for performance profile collector pods to become ready: 4
2026-04-22T18:08:51.352577816Z Gathering data for ns/openshift-insights...
2026-04-22T18:08:52.067313039Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-04-22T18:08:52.112009026Z Waiting for performance profile collector pods to become ready: 5
2026-04-22T18:08:53.242343759Z Waiting for performance profile collector pods to become ready: 6
2026-04-22T18:08:54.060793083Z Gathering data for ns/openshift-lws-operator...
2026-04-22T18:08:54.370890989Z Waiting for performance profile collector pods to become ready: 7
2026-04-22T18:08:54.600831787Z Gathering data for ns/kserve...
2026-04-22T18:08:55.504424305Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-22T18:08:55.782969546Z Collecting performance related data for node ip-10-0-133-169.ec2.internal
2026-04-22T18:08:55.790208081Z Collecting performance related data for node ip-10-0-135-36.ec2.internal
2026-04-22T18:08:55.798390600Z Collecting performance related data for node ip-10-0-143-54.ec2.internal
2026-04-22T18:08:56.454443009Z Gathering data for ns/openshift-config...
2026-04-22T18:08:56.829526615Z Gathering data for ns/openshift-config-managed...
2026-04-22T18:08:57.072578042Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-04-22T18:08:57.418501021Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-22T18:08:57.773849292Z Gathering data for ns/openshift-kube-apiserver...
2026-04-22T18:08:58.082216090Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-22T18:08:58.416973278Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-22T18:08:58.813411221Z Gathering data for ns/openshift-kube-scheduler...
2026-04-22T18:08:59.136949430Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-22T18:08:59.461039872Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-22T18:08:59.808998568Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-22T18:09:00.141223246Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-22T18:09:00.736358351Z Gathering data for ns/openshift-multus...
2026-04-22T18:09:02.077390525Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-04-22T18:09:02.196764172Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-22T18:09:03.737417829Z Gathering data for ns/openshift-host-network...
2026-04-22T18:09:04.088055843Z Gathering data for ns/openshift-network-diagnostics...
2026-04-22T18:09:04.499033378Z Collecting kubelet logs for node ip-10-0-133-169.ec2.internal
2026-04-22T18:09:04.502236600Z Collecting kubelet logs for node ip-10-0-135-36.ec2.internal
2026-04-22T18:09:04.503954616Z Collecting kubelet logs for node ip-10-0-143-54.ec2.internal
2026-04-22T18:09:04.687658186Z Gathering data for ns/openshift-network-node-identity...
2026-04-22T18:09:04.979002732Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-22T18:09:04.985152614Z INFO: Node performance data collection complete.
2026-04-22T18:09:05.176292066Z Gathering data for ns/openshift-network-operator...
2026-04-22T18:09:05.558996314Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-22T18:09:05.835967154Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-22T18:09:06.290788049Z Gathering data for ns/openshift-apiserver-operator...
2026-04-22T18:09:06.613304152Z Gathering data for ns/openshift-apiserver...
2026-04-22T18:09:07.054172679Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-22T18:09:07.082801787Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-04-22T18:09:07.324356477Z Gathering data for ns/openshift-controller-manager...
2026-04-22T18:09:07.615878074Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-22T18:09:08.281301061Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-22T18:09:08.614267590Z Gathering data for ns/openshift-service-ca-operator...
2026-04-22T18:09:08.947131771Z Gathering data for ns/openshift-service-ca...
2026-04-22T18:09:09.331326316Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-22T18:09:10.256729438Z Wrote inspect data to must-gather.
2026-04-22T18:09:10.256772794Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T18:09:10.256772794Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-22T18:09:10.398335521Z error: the server doesn't have a resource type "clusters"
2026-04-22T18:09:10.837097007Z Caches written to disk