2026-09-07T13:49:46.970025242Z [disk usage checker] Started 2026-09-07T13:49:46.973402611Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:49:47.269077513Z Gathering data for ns/openshift-cluster-version... 2026-09-07T13:49:47.512839763Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-07T13:49:47.717754153Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-07T13:49:47.843575695Z Gathering data for ns/default... 2026-09-07T13:49:48.282206934Z Gathering data for ns/openshift... 2026-09-07T13:49:48.848132818Z Gathering data for ns/kube-system... 2026-09-07T13:49:49.641265216Z Waiting on subprocesses to finish execution. 2026-09-07T13:49:49.650369265Z INFO: Gathering machine config daemon's old logs from all nodes 2026-09-07T13:49:49.651242873Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-09-07T13:49:49.651930258Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-09-07T13:49:49.660354933Z Executing Istio gather script 2026-09-07T13:49:49.665723467Z INFO: Gathering HAProxy config files 2026-09-07T13:49:49.708559273Z INFO: Collecting host service logs for crio 2026-09-07T13:49:49.708900838Z INFO: Collecting host service logs for kubelet 2026-09-07T13:49:49.709221164Z INFO: Collecting host service logs for rpm-ostreed 2026-09-07T13:49:49.709509581Z INFO: Collecting host service logs for ostree-finalize-staged 2026-09-07T13:49:49.709821957Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-09-07T13:49:49.710127253Z INFO: Collecting host service logs for machine-config-daemon-host 2026-09-07T13:49:49.710423862Z INFO: Collecting host service logs for NetworkManager 2026-09-07T13:49:49.710729317Z INFO: Collecting host service logs for openvswitch 2026-09-07T13:49:49.711086005Z INFO: Collecting host service logs for ovs-configuration 2026-09-07T13:49:49.711427559Z INFO: Collecting host service logs for ovsdb-server 2026-09-07T13:49:49.711754289Z INFO: Collecting host service logs for ovs-vswitchd 2026-09-07T13:49:49.712252257Z INFO: Waiting for worker host service log collection to complete ... 2026-09-07T13:49:49.730326582Z INFO: Waiting for node performance related collection to complete ... 2026-09-07T13:49:50.258516076Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-g5t5m 2026-09-07T13:49:50.258599640Z insights-runtime-extractor-bdthj 2026-09-07T13:49:50.258621441Z insights-runtime-extractor-lk4b9 2026-09-07T13:49:50.258640391Z insights-runtime-extractor-xsghl 2026-09-07T13:49:50.306949838Z No resources found in openshift-etcd namespace. 2026-09-07T13:49:50.602849727Z No resources found 2026-09-07T13:49:50.750925793Z Wrote inspect data to must-gather. 2026-09-07T13:49:50.863144670Z ERROR: No running kube-apiserver pods found 2026-09-07T13:49:50.882141598Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-09-07T13:49:50.882184676Z INFO: on-disk MachineConfig config collection complete. 2026-09-07T13:49:51.010986090Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-09-07T13:49:51.010986090Z See 'oc rsync -h' for help and examples 2026-09-07T13:49:51.065294206Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-09-07T13:49:51.105451212Z INFO: "metallb-operator" not detected. Skipping. 2026-09-07T13:49:51.255486695Z No resources found 2026-09-07T13:49:51.374841861Z Wrote inspect data to must-gather. 2026-09-07T13:49:51.542953564Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-09-07T13:49:51.575090073Z INFO: Waiting for HAProxy config collection to complete ... 2026-09-07T13:49:51.593300581Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-09-07T13:49:51.605234105Z Gathering data for ns/openshift-monitoring... 2026-09-07T13:49:51.627890940Z INFO: "sriov-network-operator" not detected. Skipping. 2026-09-07T13:49:51.877411958Z error: the server doesn't have a resource type "performanceprofile" 2026-09-07T13:49:51.888145576Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-07T13:49:51.906937414Z INFO: Worker host service log collection to complete. 2026-09-07T13:49:51.917872889Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.001228505Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.048510366Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.057107674Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:49:52.103057303Z INFO: Found 1 replicas - prometheus-k8s-0 2026-09-07T13:49:52.292567473Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.312292831Z Gathering data for ns/openshift-etcd... 2026-09-07T13:49:52.331609325Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.427982410Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-09-07T13:49:52.455646663Z tar: Removing leading `/' from member names 2026-09-07T13:49:52.460324285Z INFO: HAProxy config collection complete. 2026-09-07T13:49:52.460515418Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-09-07T13:49:52.547414147Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.565739777Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.579101487Z INFO: 'previous-logs' folder not found on ip-10-0-136-58.ec2.internal, skipping... 2026-09-07T13:49:52.714972883Z error: the server doesn't have a resource type "machineconfigs" 2026-09-07T13:49:52.821686072Z INFO: 'previous-logs' folder not found on ip-10-0-138-240.ec2.internal, skipping... 2026-09-07T13:49:52.834430973Z Wrote inspect data to must-gather. 2026-09-07T13:49:52.884130342Z error: the server doesn't have a resource type "machineconfigpools" 2026-09-07T13:49:53.080490492Z Wrote inspect data to must-gather. 2026-09-07T13:49:53.080533313Z error: inspection completed with the errors occurred while gathering data: 2026-09-07T13:49:53.080533313Z namespaces "assisted-installer" not found 2026-09-07T13:49:53.138091430Z INFO: 'previous-logs' folder not found on ip-10-0-141-124.ec2.internal, skipping... 2026-09-07T13:49:53.138195625Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-09-07T13:49:53.138243896Z INFO: Machine Config Daemon termination log collection complete. 2026-09-07T13:49:53.147095134Z Wrote inspect data to must-gather. 2026-09-07T13:49:53.188817747Z INFO: INTERCONNECT MODE 2026-09-07T13:49:53.188840221Z INFO: Gathering ovn-kubernetes DBs 2026-09-07T13:49:53.191250891Z INFO: Getting rules from prometheus-k8s-0 2026-09-07T13:49:53.314098293Z error: the server doesn't have a resource type "kubeletconfigs" 2026-09-07T13:49:53.369311385Z INFO: Gathering OVN_Northbound from ovnkube-node-8gjr4... 2026-09-07T13:49:53.369550636Z INFO: Gathering OVN_Northbound from ovnkube-node-phz86... 2026-09-07T13:49:53.369820388Z INFO: Gathering OVN_Northbound from ovnkube-node-sh57c... 2026-09-07T13:49:53.370105065Z INFO: Gathering OVN_Southbound from ovnkube-node-8gjr4... 2026-09-07T13:49:53.370573817Z INFO: Gathering OVN_Southbound from ovnkube-node-phz86... 2026-09-07T13:49:53.371026173Z INFO: Gathering OVN_Southbound from ovnkube-node-sh57c... 2026-09-07T13:49:53.662651949Z Wrote inspect data to must-gather. 2026-09-07T13:49:53.775183033Z tar: Removing leading `/' from member names 2026-09-07T13:49:53.794061391Z INFO: Getting status/config from prometheus-k8s-0 2026-09-07T13:49:53.799860356Z tar: Removing leading `/' from member names 2026-09-07T13:49:53.835144279Z tar: Removing leading `/' from member names 2026-09-07T13:49:53.843245789Z tar: Removing leading `/' from member names 2026-09-07T13:49:53.871562637Z tar: Removing leading `/' from member names 2026-09-07T13:49:53.894523394Z tar: Removing leading `/' from member names 2026-09-07T13:49:53.914949392Z Wrote inspect data to must-gather. 2026-09-07T13:49:54.078421694Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-09-07T13:49:54.155159164Z Wrote inspect data to must-gather. 2026-09-07T13:49:54.214750187Z INFO: Waiting for network log collection to complete ... 2026-09-07T13:49:54.214750187Z INFO: Waiting for ovnk database copies to complete ... 2026-09-07T13:49:54.215203837Z INFO: Copying ovnk databases complete. 2026-09-07T13:49:54.216892622Z 2.8M must-gather/network_logs/ovnk_database_store 2026-09-07T13:49:54.219024672Z ovnk_database_store/ 2026-09-07T13:49:54.219024672Z ovnk_database_store/ovnkube-node-phz86_nbdb 2026-09-07T13:49:54.226419227Z ovnk_database_store/ovnkube-node-8gjr4_nbdb 2026-09-07T13:49:54.232967485Z ovnk_database_store/ovnkube-node-phz86_sbdb 2026-09-07T13:49:54.249445726Z ovnk_database_store/ovnkube-node-sh57c_sbdb 2026-09-07T13:49:54.260061386Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:54.267994986Z ovnk_database_store/ovnkube-node-8gjr4_sbdb 2026-09-07T13:49:54.286670817Z ovnk_database_store/ovnkube-node-sh57c_nbdb 2026-09-07T13:49:54.295878007Z INFO: Network log collection complete. 2026-09-07T13:49:54.342777974Z INFO: Getting status/flags from prometheus-k8s-0 2026-09-07T13:49:54.379540724Z Wrote inspect data to must-gather. 2026-09-07T13:49:54.465232961Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:54.467301919Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-09-07T13:49:54.591866303Z Wrote inspect data to must-gather. 2026-09-07T13:49:54.648211773Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-09-07T13:49:54.678051296Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:54.795358218Z Wrote inspect data to must-gather. 2026-09-07T13:49:54.926320966Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:54.928646948Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-09-07T13:49:54.987999269Z Wrote inspect data to must-gather. 2026-09-07T13:49:55.130014933Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:55.219331278Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-09-07T13:49:55.269711479Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-09-07T13:49:55.272514350Z INFO: Fallback to identify the container image from release info 2026-09-07T13:49:55.332345945Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:55.334684557Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-09-07T13:49:55.553079191Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:55.595542386Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-09-07T13:49:55.764870333Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:55.767381528Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-09-07T13:49:55.814534704Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-07T13:49:55.898755082Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399 2026-09-07T13:49:55.936357732Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:56.040321642Z INFO: Getting status from alertmanager-main-0 2026-09-07T13:49:56.045299288Z daemonset.apps/perf-node-gather-daemonset created 2026-09-07T13:49:56.198612147Z Wrote inspect data to /must-gather/istio. 2026-09-07T13:49:56.208379737Z Waiting for performance profile collector pods to become ready: 1 2026-09-07T13:49:56.534630658Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-07T13:49:56.780643560Z Gathering data for ns/openshift-network-console... 2026-09-07T13:49:56.891877494Z Done executing Istio gather script 2026-09-07T13:49:57.073023922Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:49:57.332764902Z Waiting for performance profile collector pods to become ready: 2 2026-09-07T13:49:57.370021070Z Gathering data for ns/openshift-console-operator... 2026-09-07T13:49:57.888474261Z Gathering data for ns/openshift-console... 2026-09-07T13:49:58.455118913Z Waiting for performance profile collector pods to become ready: 3 2026-09-07T13:49:58.554649661Z Gathering data for ns/openshift-cluster-storage-operator... 2026-09-07T13:49:59.255010760Z Gathering data for ns/openshift-dns-operator... 2026-09-07T13:49:59.579038441Z Waiting for performance profile collector pods to become ready: 4 2026-09-07T13:49:59.666160850Z Gathering data for ns/openshift-dns... 2026-09-07T13:50:00.656037427Z Gathering data for ns/openshift-image-registry... 2026-09-07T13:50:00.695772469Z Waiting for performance profile collector pods to become ready: 5 2026-09-07T13:50:01.383685182Z Gathering data for ns/openshift-ingress-operator... 2026-09-07T13:50:01.821978181Z Waiting for performance profile collector pods to become ready: 6 2026-09-07T13:50:01.899556802Z Gathering data for ns/openshift-ingress... 2026-09-07T13:50:02.077688499Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:50:02.394557925Z Gathering data for ns/openshift-ingress-canary... 2026-09-07T13:50:02.947326578Z Waiting for performance profile collector pods to become ready: 7 2026-09-07T13:50:02.978290936Z Gathering data for ns/openshift-insights... 2026-09-07T13:50:04.066481518Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-09-07T13:50:04.307603115Z Collecting performance related data for node ip-10-0-141-124.ec2.internal 2026-09-07T13:50:04.314574989Z Collecting performance related data for node ip-10-0-138-240.ec2.internal 2026-09-07T13:50:04.322147334Z Collecting performance related data for node ip-10-0-136-58.ec2.internal 2026-09-07T13:50:05.649027982Z Gathering data for ns/openshift-jobset-operator... 2026-09-07T13:50:06.606681859Z Gathering data for ns/openshift-config... 2026-09-07T13:50:06.969853586Z Gathering data for ns/openshift-config-managed... 2026-09-07T13:50:07.082950028Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:50:07.514410420Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-09-07T13:50:07.840070109Z Gathering data for ns/openshift-kube-apiserver... 2026-09-07T13:50:08.193837953Z Gathering data for ns/openshift-kube-controller-manager... 2026-09-07T13:50:08.613018329Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-09-07T13:50:09.042976840Z Gathering data for ns/openshift-kube-scheduler... 2026-09-07T13:50:09.495993161Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-09-07T13:50:10.047675056Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-09-07T13:50:10.743159307Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-09-07T13:50:11.149514017Z Gathering data for ns/openshift-user-workload-monitoring... 2026-09-07T13:50:11.932221497Z Gathering data for ns/openshift-multus... 2026-09-07T13:50:12.088212367Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:50:12.671666551Z Collecting kubelet logs for node ip-10-0-141-124.ec2.internal 2026-09-07T13:50:12.673556189Z Collecting kubelet logs for node ip-10-0-138-240.ec2.internal 2026-09-07T13:50:12.675370857Z Collecting kubelet logs for node ip-10-0-136-58.ec2.internal 2026-09-07T13:50:12.992015606Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-gqwz6 namespace 2026-09-07T13:50:12.999215290Z INFO: Node performance data collection complete. 2026-09-07T13:50:13.839579219Z Gathering data for ns/openshift-ovn-kubernetes... 2026-09-07T13:50:15.441300544Z Gathering data for ns/openshift-host-network... 2026-09-07T13:50:15.762529789Z Gathering data for ns/openshift-network-diagnostics... 2026-09-07T13:50:16.378810421Z Gathering data for ns/openshift-network-node-identity... 2026-09-07T13:50:16.856496035Z Gathering data for ns/openshift-network-operator... 2026-09-07T13:50:17.093566899Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:50:17.372483381Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-09-07T13:50:17.719449542Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-09-07T13:50:18.380497038Z Gathering data for ns/openshift-apiserver-operator... 2026-09-07T13:50:18.773574242Z Gathering data for ns/openshift-apiserver... 2026-09-07T13:50:19.305101917Z Gathering data for ns/openshift-controller-manager-operator... 2026-09-07T13:50:19.683612139Z Gathering data for ns/openshift-controller-manager... 2026-09-07T13:50:19.991632408Z Gathering data for ns/openshift-cluster-samples-operator... 2026-09-07T13:50:20.677260452Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-09-07T13:50:21.080033459Z Gathering data for ns/openshift-service-ca-operator... 2026-09-07T13:50:21.504407805Z Gathering data for ns/openshift-service-ca... 2026-09-07T13:50:21.893533374Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-09-07T13:50:22.098730792Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-07T13:50:22.921550452Z Wrote inspect data to must-gather. 2026-09-07T13:50:22.921594058Z error: inspection completed with the errors occurred while gathering data: 2026-09-07T13:50:22.921594058Z skipping gathering secrets/support due to error: secrets "support" not found 2026-09-07T13:50:23.050046192Z error: the server doesn't have a resource type "clusters" 2026-09-07T13:50:23.231643906Z Caches written to disk