2026-04-17T21:30:25.160619911Z [disk usage checker] Started 2026-04-17T21:30:25.195648535Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 2026-04-17T21:30:25.393332144Z Gathering data for ns/openshift-cluster-version... 2026-04-17T21:30:25.567244290Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-17T21:30:25.661688366Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-17T21:30:25.762602350Z Gathering data for ns/default... 2026-04-17T21:30:26.118018377Z Gathering data for ns/openshift... 2026-04-17T21:30:26.599147607Z Gathering data for ns/kube-system... 2026-04-17T21:30:27.306394071Z Gathering data for ns/openshift-etcd... 2026-04-17T21:30:27.669531552Z Wrote inspect data to must-gather. 2026-04-17T21:30:27.669569481Z error: inspection completed with the errors occurred while gathering data: 2026-04-17T21:30:27.669569481Z namespaces "assisted-installer" not found 2026-04-17T21:30:27.781766075Z Waiting on subprocesses to finish execution. 2026-04-17T21:30:27.791030213Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-04-17T21:30:27.791430051Z INFO: Gathering HAProxy config files 2026-04-17T21:30:27.794224180Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-04-17T21:30:27.797056881Z INFO: Collecting host service logs for crio 2026-04-17T21:30:27.797397165Z INFO: Collecting host service logs for kubelet 2026-04-17T21:30:27.797769249Z INFO: Collecting host service logs for rpm-ostreed 2026-04-17T21:30:27.798130131Z INFO: Collecting host service logs for ostree-finalize-staged 2026-04-17T21:30:27.798501439Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-04-17T21:30:27.798882698Z INFO: Collecting host service logs for machine-config-daemon-host 2026-04-17T21:30:27.799233573Z INFO: Gathering machine config daemon's old logs from all nodes 2026-04-17T21:30:27.799254538Z INFO: Collecting host service logs for NetworkManager 2026-04-17T21:30:27.799792598Z INFO: Collecting host service logs for openvswitch 2026-04-17T21:30:27.800097395Z INFO: Collecting host service logs for ovs-configuration 2026-04-17T21:30:27.800405777Z INFO: Collecting host service logs for ovsdb-server 2026-04-17T21:30:27.800745615Z INFO: Collecting host service logs for ovs-vswitchd 2026-04-17T21:30:27.801143517Z INFO: Waiting for worker host service log collection to complete ... 2026-04-17T21:30:27.850552915Z INFO: Waiting for node performance related collection to complete ... 2026-04-17T21:30:28.095439031Z No resources found 2026-04-17T21:30:28.212513274Z error: the server doesn't have a resource type "performanceprofile" 2026-04-17T21:30:28.597920799Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-04-17T21:30:28.871370283Z No resources found in openshift-etcd namespace. 2026-04-17T21:30:29.011967896Z INFO: "sriov-network-operator" not detected. Skipping. 2026-04-17T21:30:29.144724229Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-04-17T21:30:29.241982926Z INFO: "metallb-operator" not detected. Skipping. 2026-04-17T21:30:29.268674239Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-04-17T21:30:29.692375005Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-d2h87 2026-04-17T21:30:29.692375005Z insights-runtime-extractor-9j7wq 2026-04-17T21:30:29.692375005Z insights-runtime-extractor-cx5kv 2026-04-17T21:30:29.692375005Z insights-runtime-extractor-rjkj7 2026-04-17T21:30:29.778742387Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-04-17T21:30:29.778831103Z INFO: on-disk MachineConfig config collection complete. 2026-04-17T21:30:29.787056488Z INFO: Found 1 replicas - prometheus-k8s-0 2026-04-17T21:30:29.886128946Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-17T21:30:30.008932403Z Wrote inspect data to must-gather. 2026-04-17T21:30:30.016148697Z INFO: Worker host service log collection to complete. 2026-04-17T21:30:30.208297027Z Wrote inspect data to must-gather. 2026-04-17T21:30:30.213115307Z No resources found 2026-04-17T21:30:30.288496397Z Wrote inspect data to must-gather. 2026-04-17T21:30:30.295786495Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 2026-04-17T21:30:30.309958335Z INFO: Waiting for HAProxy config collection to complete ... 2026-04-17T21:30:30.379356482Z Wrote inspect data to must-gather. 2026-04-17T21:30:30.380005930Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-04-17T21:30:30.380005930Z See 'oc rsync -h' for help and examples 2026-04-17T21:30:30.496572833Z ERROR: No running kube-apiserver pods found 2026-04-17T21:30:30.522716572Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-04-17T21:30:30.544640292Z Wrote inspect data to must-gather. 2026-04-17T21:30:30.583181606Z Wrote inspect data to must-gather. 2026-04-17T21:30:30.645403780Z INFO: 'previous-logs' folder not found on ip-10-0-134-198.ec2.internal, skipping... 2026-04-17T21:30:30.671414536Z error: the server doesn't have a resource type "machineconfigs" 2026-04-17T21:30:30.745682343Z tar: Removing leading `/' from member names 2026-04-17T21:30:30.751657697Z INFO: HAProxy config collection complete. 2026-04-17T21:30:30.820471463Z Gathering data for ns/kuadrant-system... 2026-04-17T21:30:31.026777899Z Wrote inspect data to must-gather. 2026-04-17T21:30:31.058627938Z Wrote inspect data to must-gather. 2026-04-17T21:30:31.084847545Z INFO: 'previous-logs' folder not found on ip-10-0-135-174.ec2.internal, skipping... 2026-04-17T21:30:31.095583219Z INFO: Getting rules from prometheus-k8s-0 2026-04-17T21:30:31.383348726Z Wrote inspect data to must-gather. 2026-04-17T21:30:31.402232999Z Wrote inspect data to must-gather. 2026-04-17T21:30:31.437380974Z INFO: 'previous-logs' folder not found on ip-10-0-138-36.ec2.internal, skipping... 2026-04-17T21:30:31.437496535Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-04-17T21:30:31.437573273Z INFO: Machine Config Daemon termination log collection complete. 2026-04-17T21:30:31.632355979Z error: the server doesn't have a resource type "kubeletconfigs" 2026-04-17T21:30:31.645718062Z INFO: Getting status/config from prometheus-k8s-0 2026-04-17T21:30:31.805256932Z Wrote inspect data to must-gather. 2026-04-17T21:30:31.819980818Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-17T21:30:31.930840417Z Wrote inspect data to must-gather. 2026-04-17T21:30:32.137623518Z INFO: Getting status/flags from prometheus-k8s-0 2026-04-17T21:30:32.187236468Z INFO: INTERCONNECT MODE 2026-04-17T21:30:32.187321968Z INFO: Gathering ovn-kubernetes DBs 2026-04-17T21:30:32.256170532Z Wrote inspect data to must-gather. 2026-04-17T21:30:32.397841103Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-17T21:30:32.418432045Z INFO: Gathering OVN_Northbound from ovnkube-node-7q89l... 2026-04-17T21:30:32.418763797Z INFO: Gathering OVN_Northbound from ovnkube-node-hn2f5... 2026-04-17T21:30:32.419035104Z INFO: Gathering OVN_Northbound from ovnkube-node-wvwmf... 2026-04-17T21:30:32.419609647Z INFO: Gathering OVN_Southbound from ovnkube-node-7q89l... 2026-04-17T21:30:32.419881333Z INFO: Gathering OVN_Southbound from ovnkube-node-hn2f5... 2026-04-17T21:30:32.420134778Z INFO: Gathering OVN_Southbound from ovnkube-node-wvwmf... 2026-04-17T21:30:32.460005732Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-04-17T21:30:32.717710722Z tar: Removing leading `/' from member names 2026-04-17T21:30:32.785078265Z Wrote inspect data to must-gather. 2026-04-17T21:30:32.861063764Z tar: Removing leading `/' from member names 2026-04-17T21:30:32.872802890Z Gathering data for ns/openshift-monitoring... 2026-04-17T21:30:32.940498971Z tar: Removing leading `/' from member names 2026-04-17T21:30:32.952818382Z tar: Removing leading `/' from member names 2026-04-17T21:30:32.952957341Z tar: Removing leading `/' from member names 2026-04-17T21:30:32.956434243Z tar: Removing leading `/' from member names 2026-04-17T21:30:33.043900197Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-04-17T21:30:33.211329770Z Wrote inspect data to must-gather. 2026-04-17T21:30:33.359534457Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-04-17T21:30:33.560942113Z INFO: Waiting for network log collection to complete ... 2026-04-17T21:30:33.561071443Z INFO: Waiting for ovnk database copies to complete ... 2026-04-17T21:30:33.561607722Z INFO: Copying ovnk databases complete. 2026-04-17T21:30:33.563255796Z 4.0M must-gather/network_logs/ovnk_database_store 2026-04-17T21:30:33.565240262Z ovnk_database_store/ 2026-04-17T21:30:33.565240262Z ovnk_database_store/ovnkube-node-7q89l_nbdb 2026-04-17T21:30:33.575645571Z ovnk_database_store/ovnkube-node-7q89l_sbdb 2026-04-17T21:30:33.586112201Z Wrote inspect data to must-gather. 2026-04-17T21:30:33.601431122Z ovnk_database_store/ovnkube-node-hn2f5_sbdb 2026-04-17T21:30:33.632351789Z ovnk_database_store/ovnkube-node-wvwmf_sbdb 2026-04-17T21:30:33.655312556Z ovnk_database_store/ovnkube-node-wvwmf_nbdb 2026-04-17T21:30:33.664685480Z ovnk_database_store/ovnkube-node-hn2f5_nbdb 2026-04-17T21:30:33.680801727Z INFO: Network log collection complete. 2026-04-17T21:30:33.781233782Z INFO: Getting status from alertmanager-main-0 2026-04-17T21:30:33.884322724Z Wrote inspect data to must-gather. 2026-04-17T21:30:34.190844213Z Wrote inspect data to must-gather. 2026-04-17T21:30:34.450664665Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-04-17T21:30:34.453576405Z INFO: Fallback to identify the container image from release info 2026-04-17T21:30:35.035129026Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d 2026-04-17T21:30:35.053856013Z Gathering data for ns/openshift-network-console... 2026-04-17T21:30:35.194056872Z daemonset.apps/perf-node-gather-daemonset created 2026-04-17T21:30:35.304990913Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 2026-04-17T21:30:35.345704167Z Waiting for performance profile collector pods to become ready: 1 2026-04-17T21:30:35.589850335Z Gathering data for ns/openshift-console-operator... 2026-04-17T21:30:36.060016585Z Gathering data for ns/openshift-console... 2026-04-17T21:30:36.519024488Z Waiting for performance profile collector pods to become ready: 2 2026-04-17T21:30:36.621992341Z Gathering data for ns/openshift-cluster-storage-operator... 2026-04-17T21:30:37.053147866Z Gathering data for ns/openshift-dns-operator... 2026-04-17T21:30:37.474718975Z Gathering data for ns/openshift-dns... 2026-04-17T21:30:37.494024801Z Wrote inspect data to must-gather. 2026-04-17T21:30:37.634294128Z Waiting for performance profile collector pods to become ready: 3 2026-04-17T21:30:38.216991288Z Gathering data for ns/openshift-image-registry... 2026-04-17T21:30:38.725975204Z Gathering data for ns/openshift-ingress-operator... 2026-04-17T21:30:38.764337532Z Waiting for performance profile collector pods to become ready: 4 2026-04-17T21:30:39.081191730Z Gathering data for ns/openshift-ingress... 2026-04-17T21:30:39.753210351Z Gathering data for ns/openshift-ingress-canary... 2026-04-17T21:30:39.893866079Z Waiting for performance profile collector pods to become ready: 5 2026-04-17T21:30:40.268006609Z Gathering data for ns/openshift-insights... 2026-04-17T21:30:40.310300153Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 2026-04-17T21:30:41.015673016Z Waiting for performance profile collector pods to become ready: 6 2026-04-17T21:30:42.139289446Z Waiting for performance profile collector pods to become ready: 7 2026-04-17T21:30:42.607047422Z Gathering data for ns/opendatahub... 2026-04-17T21:30:43.260772895Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-04-17T21:30:43.534017694Z Collecting performance related data for node ip-10-0-134-198.ec2.internal 2026-04-17T21:30:43.545658186Z Collecting performance related data for node ip-10-0-135-174.ec2.internal 2026-04-17T21:30:43.552688822Z Collecting performance related data for node ip-10-0-138-36.ec2.internal 2026-04-17T21:30:44.166021731Z Gathering data for ns/openshift-lws-operator... 2026-04-17T21:30:45.315216468Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 2026-04-17T21:30:45.562548825Z Gathering data for ns/openshift-config... 2026-04-17T21:30:45.919773898Z Gathering data for ns/openshift-config-managed... 2026-04-17T21:30:46.450626467Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-04-17T21:30:46.823111749Z Gathering data for ns/openshift-kube-apiserver... 2026-04-17T21:30:47.146838762Z Gathering data for ns/openshift-kube-controller-manager... 2026-04-17T21:30:47.499676986Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-04-17T21:30:47.841222995Z Gathering data for ns/openshift-kube-scheduler... 2026-04-17T21:30:48.147623996Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-04-17T21:30:48.543653617Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-04-17T21:30:48.901599832Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-04-17T21:30:49.254572095Z Gathering data for ns/openshift-user-workload-monitoring... 2026-04-17T21:30:49.838022699Z Gathering data for ns/openshift-multus... 2026-04-17T21:30:50.320486699Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 2026-04-17T21:30:51.291604682Z Gathering data for ns/openshift-ovn-kubernetes... 2026-04-17T21:30:52.286956096Z Collecting kubelet logs for node ip-10-0-134-198.ec2.internal 2026-04-17T21:30:52.292310277Z Collecting kubelet logs for node ip-10-0-135-174.ec2.internal 2026-04-17T21:30:52.294578251Z Collecting kubelet logs for node ip-10-0-138-36.ec2.internal 2026-04-17T21:30:52.661791091Z daemonset.apps "perf-node-gather-daemonset" deleted 2026-04-17T21:30:52.666773802Z INFO: Node performance data collection complete. 2026-04-17T21:30:53.058673203Z Gathering data for ns/openshift-host-network... 2026-04-17T21:30:53.406028015Z Gathering data for ns/openshift-network-diagnostics... 2026-04-17T21:30:53.972619802Z Gathering data for ns/openshift-network-node-identity... 2026-04-17T21:30:54.404804725Z Gathering data for ns/openshift-network-operator... 2026-04-17T21:30:54.796379149Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-04-17T21:30:55.101740807Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-04-17T21:30:55.325080457Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70 2026-04-17T21:30:55.575687228Z Gathering data for ns/openshift-apiserver-operator... 2026-04-17T21:30:55.885095399Z Gathering data for ns/openshift-apiserver... 2026-04-17T21:30:56.346129195Z Gathering data for ns/openshift-controller-manager-operator... 2026-04-17T21:30:56.623534583Z Gathering data for ns/openshift-controller-manager... 2026-04-17T21:30:56.960625353Z Gathering data for ns/openshift-cluster-samples-operator... 2026-04-17T21:30:57.629774226Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-04-17T21:30:57.950119329Z Gathering data for ns/openshift-service-ca-operator... 2026-04-17T21:30:58.286545621Z Gathering data for ns/openshift-service-ca... 2026-04-17T21:30:58.680491244Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-04-17T21:30:59.699803028Z Wrote inspect data to must-gather. 2026-04-17T21:30:59.699848537Z error: inspection completed with the errors occurred while gathering data: 2026-04-17T21:30:59.699848537Z skipping gathering secrets/support due to error: secrets "support" not found 2026-04-17T21:30:59.833590439Z error: the server doesn't have a resource type "clusters" 2026-04-17T21:31:00.198339097Z Caches written to disk