2026-05-26T22:19:41.924925930Z [disk usage checker] Started
2026-05-26T22:19:41.928600520Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:19:42.148650660Z Gathering data for ns/openshift-cluster-version...
2026-05-26T22:19:42.319982923Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-26T22:19:42.414816616Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-26T22:19:42.517630825Z Gathering data for ns/default...
2026-05-26T22:19:42.861726658Z Gathering data for ns/openshift...
2026-05-26T22:19:43.343761654Z Gathering data for ns/kube-system...
2026-05-26T22:19:44.089219829Z Gathering data for ns/openshift-etcd...
2026-05-26T22:19:44.452404597Z Wrote inspect data to must-gather.
2026-05-26T22:19:44.452438667Z error: inspection completed with the errors occurred while gathering data:
2026-05-26T22:19:44.452438667Z     namespaces "assisted-installer" not found
2026-05-26T22:19:44.480473838Z Waiting on subprocesses to finish execution.
2026-05-26T22:19:44.487345353Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-26T22:19:44.493542039Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-26T22:19:44.495002932Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-26T22:19:44.496940323Z INFO: Collecting host service logs for crio
2026-05-26T22:19:44.497245524Z INFO: Collecting host service logs for kubelet
2026-05-26T22:19:44.497673463Z INFO: Collecting host service logs for rpm-ostreed
2026-05-26T22:19:44.498012271Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-26T22:19:44.498405067Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-26T22:19:44.498757674Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-26T22:19:44.499313232Z INFO: Collecting host service logs for NetworkManager
2026-05-26T22:19:44.499640846Z INFO: Waiting for node performance related collection to complete ...
2026-05-26T22:19:44.499663763Z INFO: Collecting host service logs for openvswitch
2026-05-26T22:19:44.499965157Z INFO: Collecting host service logs for ovs-configuration
2026-05-26T22:19:44.500308151Z INFO: Collecting host service logs for ovsdb-server
2026-05-26T22:19:44.500660315Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-26T22:19:44.501068988Z INFO: Waiting for worker host service log collection to complete ...
2026-05-26T22:19:44.506197654Z INFO: Gathering HAProxy config files
2026-05-26T22:19:45.211582058Z No resources found
2026-05-26T22:19:45.272767052Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-26T22:19:45.272798586Z INFO: on-disk MachineConfig config collection complete.
2026-05-26T22:19:45.519985056Z INFO: "metallb-operator" not detected. Skipping.
2026-05-26T22:19:45.540501100Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-26T22:19:45.663209083Z No resources found in openshift-etcd namespace.
2026-05-26T22:19:45.733363173Z Wrote inspect data to must-gather.
2026-05-26T22:19:45.777457026Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-26T22:19:45.790722158Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-p47pq
2026-05-26T22:19:45.790785817Z insights-runtime-extractor-6xbqc
2026-05-26T22:19:45.790807797Z insights-runtime-extractor-hdrmw
2026-05-26T22:19:45.790844904Z insights-runtime-extractor-rsmk9
2026-05-26T22:19:45.911924066Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-26T22:19:46.030906856Z INFO: Worker host service log collection to complete.
2026-05-26T22:19:46.032421755Z error: the server doesn't have a resource type "performanceprofile"
2026-05-26T22:19:46.288916734Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-26T22:19:46.580876093Z ERROR: No running kube-apiserver pods found
2026-05-26T22:19:46.635258795Z No resources found
2026-05-26T22:19:46.802442851Z Wrote inspect data to must-gather.
2026-05-26T22:19:46.840523536Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-26T22:19:46.847374410Z Wrote inspect data to must-gather.
2026-05-26T22:19:46.873864070Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-26T22:19:46.873864070Z See 'oc rsync -h' for help and examples
2026-05-26T22:19:46.951359947Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:19:46.978830302Z Wrote inspect data to must-gather.
2026-05-26T22:19:47.027483265Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-26T22:19:47.075952884Z Wrote inspect data to must-gather.
2026-05-26T22:19:47.081177050Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-26T22:19:47.162071387Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-26T22:19:47.178105382Z INFO: 'previous-logs' folder not found on ip-10-0-128-18.ec2.internal, skipping...
2026-05-26T22:19:47.281592485Z tar: Removing leading `/' from member names
2026-05-26T22:19:47.288813382Z INFO: HAProxy config collection complete.
2026-05-26T22:19:47.365925528Z Wrote inspect data to must-gather.
2026-05-26T22:19:47.472356952Z Gathering data for ns/kuadrant-system...
2026-05-26T22:19:47.554998466Z Wrote inspect data to must-gather.
2026-05-26T22:19:47.614037584Z INFO: 'previous-logs' folder not found on ip-10-0-138-198.ec2.internal, skipping...
2026-05-26T22:19:47.636392466Z error: the server doesn't have a resource type "machineconfigs"
2026-05-26T22:19:47.770724590Z INFO: Getting rules from prometheus-k8s-0
2026-05-26T22:19:47.957890119Z Wrote inspect data to must-gather.
2026-05-26T22:19:47.962734602Z INFO: 'previous-logs' folder not found on ip-10-0-140-172.ec2.internal, skipping...
2026-05-26T22:19:47.962837363Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-26T22:19:47.962905288Z INFO: Machine Config Daemon termination log collection complete.
2026-05-26T22:19:47.988688126Z Wrote inspect data to must-gather.
2026-05-26T22:19:48.251219887Z Wrote inspect data to must-gather.
2026-05-26T22:19:48.321432494Z INFO: Getting status/config from prometheus-k8s-0
2026-05-26T22:19:48.335797631Z Wrote inspect data to must-gather.
2026-05-26T22:19:48.488694147Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-26T22:19:48.771034750Z INFO: INTERCONNECT MODE
2026-05-26T22:19:48.773743922Z INFO: Gathering ovn-kubernetes DBs
2026-05-26T22:19:48.829235224Z Wrote inspect data to must-gather.
2026-05-26T22:19:48.879388294Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-26T22:19:48.914805277Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-26T22:19:48.988364402Z INFO: Gathering OVN_Northbound from ovnkube-node-66skj...
2026-05-26T22:19:48.988669517Z INFO: Gathering OVN_Northbound from ovnkube-node-ct5l7...
2026-05-26T22:19:48.988970089Z INFO: Gathering OVN_Northbound from ovnkube-node-scdhw...
2026-05-26T22:19:48.989272080Z INFO: Gathering OVN_Southbound from ovnkube-node-66skj...
2026-05-26T22:19:48.989605524Z INFO: Gathering OVN_Southbound from ovnkube-node-ct5l7...
2026-05-26T22:19:48.989888912Z INFO: Gathering OVN_Southbound from ovnkube-node-scdhw...
2026-05-26T22:19:49.172853774Z Wrote inspect data to must-gather.
2026-05-26T22:19:49.308282403Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-26T22:19:49.353071181Z tar: Removing leading `/' from member names
2026-05-26T22:19:49.427430467Z tar: Removing leading `/' from member names
2026-05-26T22:19:49.490734188Z tar: Removing leading `/' from member names
2026-05-26T22:19:49.514139333Z tar: Removing leading `/' from member names
2026-05-26T22:19:49.527279091Z tar: Removing leading `/' from member names
2026-05-26T22:19:49.543578018Z tar: Removing leading `/' from member names
2026-05-26T22:19:49.668583243Z W0526 22:19:49.668499    1280 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-26T22:19:49.698294446Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-26T22:19:49.702427881Z Wrote inspect data to must-gather.
2026-05-26T22:19:49.767895701Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-26T22:19:49.988916949Z Gathering data for ns/openshift-monitoring...
2026-05-26T22:19:50.122834597Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-26T22:19:50.134046316Z Wrote inspect data to must-gather.
2026-05-26T22:19:50.190976760Z INFO: Waiting for network log collection to complete ...
2026-05-26T22:19:50.191026448Z INFO: Waiting for ovnk database copies to complete ...
2026-05-26T22:19:50.191556196Z INFO: Copying ovnk databases complete.
2026-05-26T22:19:50.193688111Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-05-26T22:19:50.196445757Z ovnk_database_store/
2026-05-26T22:19:50.196445757Z ovnk_database_store/ovnkube-node-scdhw_nbdb
2026-05-26T22:19:50.209355744Z ovnk_database_store/ovnkube-node-66skj_sbdb
2026-05-26T22:19:50.235366212Z ovnk_database_store/ovnkube-node-ct5l7_nbdb
2026-05-26T22:19:50.248701048Z ovnk_database_store/ovnkube-node-scdhw_sbdb
2026-05-26T22:19:50.275960461Z ovnk_database_store/ovnkube-node-66skj_nbdb
2026-05-26T22:19:50.292268758Z ovnk_database_store/ovnkube-node-ct5l7_sbdb
2026-05-26T22:19:50.321595821Z INFO: Network log collection complete.
2026-05-26T22:19:50.485131351Z Wrote inspect data to must-gather.
2026-05-26T22:19:50.569025838Z INFO: Getting status from alertmanager-main-0
2026-05-26T22:19:50.790876425Z Wrote inspect data to must-gather.
2026-05-26T22:19:51.089903165Z Wrote inspect data to must-gather.
2026-05-26T22:19:51.376749979Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-26T22:19:51.379345788Z INFO: Fallback to identify the container image from release info
2026-05-26T22:19:51.921352639Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-26T22:19:51.958867389Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:19:52.079104498Z daemonset.apps/perf-node-gather-daemonset created
2026-05-26T22:19:52.198936258Z Gathering data for ns/openshift-network-console...
2026-05-26T22:19:52.235335569Z Waiting for performance profile collector pods to become ready: 1
2026-05-26T22:19:52.764750541Z Gathering data for ns/openshift-console-operator...
2026-05-26T22:19:53.204800442Z Gathering data for ns/openshift-console...
2026-05-26T22:19:53.380448980Z Waiting for performance profile collector pods to become ready: 2
2026-05-26T22:19:53.771248379Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-26T22:19:54.195609792Z Gathering data for ns/openshift-dns-operator...
2026-05-26T22:19:54.419758529Z Wrote inspect data to must-gather.
2026-05-26T22:19:54.520125352Z Waiting for performance profile collector pods to become ready: 3
2026-05-26T22:19:54.574862976Z Gathering data for ns/openshift-dns...
2026-05-26T22:19:55.194989281Z Gathering data for ns/openshift-image-registry...
2026-05-26T22:19:55.661377612Z Waiting for performance profile collector pods to become ready: 4
2026-05-26T22:19:55.714108967Z Gathering data for ns/openshift-ingress-operator...
2026-05-26T22:19:56.076685090Z Gathering data for ns/openshift-ingress...
2026-05-26T22:19:56.792860485Z Waiting for performance profile collector pods to become ready: 5
2026-05-26T22:19:56.901540360Z Gathering data for ns/openshift-ingress-canary...
2026-05-26T22:19:56.964308641Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:19:57.379599915Z Gathering data for ns/openshift-insights...
2026-05-26T22:19:57.918715499Z Waiting for performance profile collector pods to become ready: 6
2026-05-26T22:19:59.036175699Z Waiting for performance profile collector pods to become ready: 7
2026-05-26T22:19:59.373951028Z Gathering data for ns/opendatahub...
2026-05-26T22:20:00.173338444Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-26T22:20:00.454211927Z Collecting performance related data for node ip-10-0-140-172.ec2.internal
2026-05-26T22:20:00.461505342Z Collecting performance related data for node ip-10-0-138-198.ec2.internal
2026-05-26T22:20:00.469693161Z Collecting performance related data for node ip-10-0-128-18.ec2.internal
2026-05-26T22:20:00.996326646Z Gathering data for ns/openshift-lws-operator...
2026-05-26T22:20:01.969231534Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:20:02.440284345Z Gathering data for ns/openshift-config...
2026-05-26T22:20:02.755377755Z Gathering data for ns/openshift-config-managed...
2026-05-26T22:20:03.303591525Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-26T22:20:03.672257450Z Gathering data for ns/openshift-kube-apiserver...
2026-05-26T22:20:04.017122487Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-26T22:20:04.355694310Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-26T22:20:04.719274857Z Gathering data for ns/openshift-kube-scheduler...
2026-05-26T22:20:05.040403260Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-26T22:20:05.392258334Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-26T22:20:05.770206749Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-26T22:20:06.131450947Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-26T22:20:06.751459495Z Gathering data for ns/openshift-multus...
2026-05-26T22:20:06.974236269Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:20:08.279147915Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-26T22:20:09.367401600Z Collecting kubelet logs for node ip-10-0-140-172.ec2.internal
2026-05-26T22:20:09.369128700Z Collecting kubelet logs for node ip-10-0-138-198.ec2.internal
2026-05-26T22:20:09.370879147Z Collecting kubelet logs for node ip-10-0-128-18.ec2.internal
2026-05-26T22:20:09.776169956Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-26T22:20:09.783098333Z INFO: Node performance data collection complete.
2026-05-26T22:20:10.008581651Z Gathering data for ns/openshift-host-network...
2026-05-26T22:20:10.357278758Z Gathering data for ns/openshift-network-diagnostics...
2026-05-26T22:20:10.920345430Z Gathering data for ns/openshift-network-node-identity...
2026-05-26T22:20:11.383588811Z Gathering data for ns/openshift-network-operator...
2026-05-26T22:20:11.831785863Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-26T22:20:11.979927797Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:20:12.129862555Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-26T22:20:12.631654604Z Gathering data for ns/openshift-apiserver-operator...
2026-05-26T22:20:12.964135549Z Gathering data for ns/openshift-apiserver...
2026-05-26T22:20:13.473177057Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-26T22:20:13.845078811Z Gathering data for ns/openshift-controller-manager...
2026-05-26T22:20:14.172144161Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-26T22:20:14.850988904Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-26T22:20:15.190392214Z Gathering data for ns/openshift-service-ca-operator...
2026-05-26T22:20:15.534367043Z Gathering data for ns/openshift-service-ca...
2026-05-26T22:20:15.957497044Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-26T22:20:16.984822958Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-26T22:20:17.044158836Z Wrote inspect data to must-gather.
2026-05-26T22:20:17.044208963Z error: inspection completed with the errors occurred while gathering data:
2026-05-26T22:20:17.044208963Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-26T22:20:17.190778385Z error: the server doesn't have a resource type "clusters"
2026-05-26T22:20:17.371374586Z Caches written to disk