2026-07-06T20:58:06.717599371Z [disk usage checker] Started 2026-07-06T20:58:06.720892074Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T20:58:06.915000786Z Gathering data for ns/openshift-cluster-version... 2026-07-06T20:58:07.061102467Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-06T20:58:07.173916804Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T20:58:07.281184275Z Gathering data for ns/default... 2026-07-06T20:58:07.632793356Z Gathering data for ns/openshift... 2026-07-06T20:58:08.123515948Z Gathering data for ns/kube-system... 2026-07-06T20:58:09.173595112Z Gathering data for ns/openshift-etcd... 2026-07-06T20:58:09.548983714Z Wrote inspect data to must-gather. 2026-07-06T20:58:09.549014445Z error: inspection completed with the errors occurred while gathering data: 2026-07-06T20:58:09.549014445Z namespaces "assisted-installer" not found 2026-07-06T20:58:09.561511916Z Waiting on subprocesses to finish execution. 2026-07-06T20:58:09.566309783Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-06T20:58:09.570075587Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-06T20:58:09.571319251Z INFO: Gathering HAProxy config files 2026-07-06T20:58:09.577718984Z INFO: Collecting host service logs for crio 2026-07-06T20:58:09.578089061Z INFO: Collecting host service logs for kubelet 2026-07-06T20:58:09.578471296Z INFO: Collecting host service logs for rpm-ostreed 2026-07-06T20:58:09.578797732Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-06T20:58:09.579682732Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-06T20:58:09.580046228Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-06T20:58:09.580816981Z INFO: Collecting host service logs for NetworkManager 2026-07-06T20:58:09.581579446Z INFO: Collecting host service logs for openvswitch 2026-07-06T20:58:09.581964613Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-06T20:58:09.582379149Z INFO: Collecting host service logs for ovs-configuration 2026-07-06T20:58:09.582690072Z INFO: Collecting host service logs for ovsdb-server 2026-07-06T20:58:09.582995526Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-06T20:58:09.584706890Z INFO: Waiting for worker host service log collection to complete ... 2026-07-06T20:58:09.627032411Z Executing Istio gather script 2026-07-06T20:58:09.665336593Z INFO: Waiting for node performance related collection to complete ... 2026-07-06T20:58:10.322750501Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-06T20:58:10.344131614Z No resources found in openshift-etcd namespace. 2026-07-06T20:58:10.513473276Z INFO: "metallb-operator" not detected. Skipping. 2026-07-06T20:58:10.544501662Z Wrote inspect data to must-gather. 2026-07-06T20:58:10.658440632Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-9whmv 2026-07-06T20:58:10.658494776Z insights-runtime-extractor-ghpnj 2026-07-06T20:58:10.658514594Z insights-runtime-extractor-gtj4z 2026-07-06T20:58:10.658533145Z insights-runtime-extractor-vxbs2 2026-07-06T20:58:10.771586580Z No resources found 2026-07-06T20:58:10.844003802Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-06T20:58:11.087592289Z Wrote inspect data to must-gather. 2026-07-06T20:58:11.125245682Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-06T20:58:11.125397122Z INFO: on-disk MachineConfig config collection complete. 2026-07-06T20:58:11.201996988Z Wrote inspect data to must-gather. 2026-07-06T20:58:11.272968990Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-06T20:58:11.275562152Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-06T20:58:11.320796629Z Wrote inspect data to must-gather. 2026-07-06T20:58:11.358424104Z error: the server doesn't have a resource type "performanceprofile" 2026-07-06T20:58:11.619436057Z INFO: Worker host service log collection to complete. 2026-07-06T20:58:11.629979279Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T20:58:11.744639882Z No resources found 2026-07-06T20:58:11.751152154Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-06T20:58:11.751152154Z See 'oc rsync -h' for help and examples 2026-07-06T20:58:11.765112055Z Wrote inspect data to must-gather. 2026-07-06T20:58:11.765626763Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-06T20:58:11.794427796Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T20:58:11.918953260Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-06T20:58:11.960844418Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-06T20:58:11.986608105Z ERROR: No running kube-apiserver pods found 2026-07-06T20:58:12.001689562Z Wrote inspect data to must-gather. 2026-07-06T20:58:12.007022916Z Wrote inspect data to must-gather. 2026-07-06T20:58:12.099869358Z Gathering data for ns/openshift-monitoring... 2026-07-06T20:58:12.106561258Z INFO: 'previous-logs' folder not found on ip-10-0-135-232.ec2.internal, skipping... 2026-07-06T20:58:12.253261169Z tar: Removing leading `/' from member names 2026-07-06T20:58:12.259109207Z INFO: HAProxy config collection complete. 2026-07-06T20:58:12.267103992Z error: the server doesn't have a resource type "machineconfigs" 2026-07-06T20:58:12.339918860Z Wrote inspect data to must-gather. 2026-07-06T20:58:12.439014510Z INFO: 'previous-logs' folder not found on ip-10-0-138-211.ec2.internal, skipping... 2026-07-06T20:58:12.464579397Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-06T20:58:12.516469644Z INFO: Getting rules from prometheus-k8s-0 2026-07-06T20:58:12.682072485Z Wrote inspect data to must-gather. 2026-07-06T20:58:12.737986852Z Wrote inspect data to must-gather. 2026-07-06T20:58:12.773052848Z INFO: 'previous-logs' folder not found on ip-10-0-141-189.ec2.internal, skipping... 2026-07-06T20:58:12.773085625Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-06T20:58:12.773093177Z INFO: Machine Config Daemon termination log collection complete. 2026-07-06T20:58:12.903112262Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-06T20:58:12.937380055Z Wrote inspect data to must-gather. 2026-07-06T20:58:13.062108912Z INFO: Getting status/config from prometheus-k8s-0 2026-07-06T20:58:13.151361519Z Wrote inspect data to must-gather. 2026-07-06T20:58:13.260103412Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-06T20:58:13.266899629Z INFO: INTERCONNECT MODE 2026-07-06T20:58:13.266927858Z INFO: Gathering ovn-kubernetes DBs 2026-07-06T20:58:13.379505453Z Wrote inspect data to must-gather. 2026-07-06T20:58:13.444366400Z INFO: Gathering OVN_Northbound from ovnkube-node-6d7c9... 2026-07-06T20:58:13.444605663Z INFO: Gathering OVN_Northbound from ovnkube-node-gth2m... 2026-07-06T20:58:13.444847217Z INFO: Gathering OVN_Northbound from ovnkube-node-s88v7... 2026-07-06T20:58:13.445122678Z INFO: Gathering OVN_Southbound from ovnkube-node-6d7c9... 2026-07-06T20:58:13.445438236Z INFO: Gathering OVN_Southbound from ovnkube-node-gth2m... 2026-07-06T20:58:13.445704364Z INFO: Gathering OVN_Southbound from ovnkube-node-s88v7... 2026-07-06T20:58:13.575709341Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-06T20:58:13.738720687Z Wrote inspect data to must-gather. 2026-07-06T20:58:13.752977068Z tar: Removing leading `/' from member names 2026-07-06T20:58:13.805475085Z tar: Removing leading `/' from member names 2026-07-06T20:58:13.808601794Z tar: Removing leading `/' from member names 2026-07-06T20:58:13.841321787Z tar: Removing leading `/' from member names 2026-07-06T20:58:13.854917411Z tar: Removing leading `/' from member names 2026-07-06T20:58:13.867496840Z tar: Removing leading `/' from member names 2026-07-06T20:58:13.965723763Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-06T20:58:13.979521609Z Wrote inspect data to must-gather. 2026-07-06T20:58:14.209268818Z Wrote inspect data to must-gather. 2026-07-06T20:58:14.258515893Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-06T20:58:14.282341716Z INFO: Waiting for network log collection to complete ... 2026-07-06T20:58:14.282395648Z INFO: Waiting for ovnk database copies to complete ... 2026-07-06T20:58:14.282892215Z INFO: Copying ovnk databases complete. 2026-07-06T20:58:14.284445132Z 3.1M must-gather/network_logs/ovnk_database_store 2026-07-06T20:58:14.286589684Z ovnk_database_store/ 2026-07-06T20:58:14.286637682Z ovnk_database_store/ovnkube-node-6d7c9_sbdb 2026-07-06T20:58:14.304905873Z ovnk_database_store/ovnkube-node-gth2m_sbdb 2026-07-06T20:58:14.322657681Z ovnk_database_store/ovnkube-node-6d7c9_nbdb 2026-07-06T20:58:14.331002697Z ovnk_database_store/ovnkube-node-gth2m_nbdb 2026-07-06T20:58:14.338895563Z ovnk_database_store/ovnkube-node-s88v7_nbdb 2026-07-06T20:58:14.347080598Z ovnk_database_store/ovnkube-node-s88v7_sbdb 2026-07-06T20:58:14.367559433Z INFO: Network log collection complete. 2026-07-06T20:58:14.406276563Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-06T20:58:14.427562844Z Wrote inspect data to must-gather. 2026-07-06T20:58:14.564007540Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-06T20:58:14.605310973Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:14.659352305Z Wrote inspect data to must-gather. 2026-07-06T20:58:14.692308473Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-06T20:58:14.812218658Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:14.815291193Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-06T20:58:14.963469066Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-06T20:58:14.966282318Z INFO: Fallback to identify the container image from release info 2026-07-06T20:58:14.993104661Z INFO: Getting status from alertmanager-main-0 2026-07-06T20:58:15.041998243Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:15.237216521Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:15.239518988Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-06T20:58:15.319487253Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T20:58:15.432558012Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:15.497885054Z Gathering data for ns/openshift-network-console... 2026-07-06T20:58:15.572963102Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a 2026-07-06T20:58:15.655069722Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:15.657557780Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-06T20:58:15.732584814Z daemonset.apps/perf-node-gather-daemonset created 2026-07-06T20:58:15.881385400Z Waiting for performance profile collector pods to become ready: 1 2026-07-06T20:58:15.906978142Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:16.048753866Z Gathering data for ns/openshift-console-operator... 2026-07-06T20:58:16.102710208Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:16.106618072Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-06T20:58:16.301883855Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:16.528080758Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:16.530758927Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-06T20:58:16.538999597Z Gathering data for ns/openshift-console... 2026-07-06T20:58:16.693714580Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:16.802554052Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T20:58:16.898571318Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:16.901064914Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-06T20:58:17.048076150Z Waiting for performance profile collector pods to become ready: 2 2026-07-06T20:58:17.099806996Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:17.311819948Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:17.314317945Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-06T20:58:17.329722925Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-06T20:58:17.488317750Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:17.673403519Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:17.675829487Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-06T20:58:17.851507105Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:18.026513717Z Gathering data for ns/openshift-dns-operator... 2026-07-06T20:58:18.078374159Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:18.081917147Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-06T20:58:18.196274584Z Waiting for performance profile collector pods to become ready: 3 2026-07-06T20:58:18.254470096Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:18.417604894Z Gathering data for ns/openshift-dns... 2026-07-06T20:58:18.462532693Z Wrote inspect data to /must-gather/istio. 2026-07-06T20:58:19.142011339Z Done executing Istio gather script 2026-07-06T20:58:19.277185535Z Gathering data for ns/openshift-image-registry... 2026-07-06T20:58:19.339556036Z Waiting for performance profile collector pods to become ready: 4 2026-07-06T20:58:19.897480966Z Gathering data for ns/openshift-ingress-operator... 2026-07-06T20:58:20.279653204Z Gathering data for ns/openshift-ingress... 2026-07-06T20:58:20.475847129Z Waiting for performance profile collector pods to become ready: 5 2026-07-06T20:58:20.701438019Z Gathering data for ns/openshift-ingress-canary... 2026-07-06T20:58:21.111389948Z Gathering data for ns/openshift-insights... 2026-07-06T20:58:21.587891034Z Waiting for performance profile collector pods to become ready: 6 2026-07-06T20:58:21.807370004Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T20:58:22.713320349Z Waiting for performance profile collector pods to become ready: 7 2026-07-06T20:58:23.466373532Z Gathering data for ns/kserve... 2026-07-06T20:58:23.833251609Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-06T20:58:24.091733336Z Collecting performance related data for node ip-10-0-135-232.ec2.internal 2026-07-06T20:58:24.098358276Z Collecting performance related data for node ip-10-0-138-211.ec2.internal 2026-07-06T20:58:24.106116219Z Collecting performance related data for node ip-10-0-141-189.ec2.internal 2026-07-06T20:58:25.269165067Z Gathering data for ns/openshift-config... 2026-07-06T20:58:25.746946625Z Gathering data for ns/openshift-config-managed... 2026-07-06T20:58:26.233147337Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-06T20:58:26.550681632Z Gathering data for ns/openshift-kube-apiserver... 2026-07-06T20:58:26.813551362Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T20:58:26.934182368Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-06T20:58:27.294100027Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-06T20:58:27.719672075Z Gathering data for ns/openshift-kube-scheduler... 2026-07-06T20:58:28.029604524Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-06T20:58:28.351115285Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-06T20:58:28.733938386Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-06T20:58:29.062709110Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-06T20:58:29.775689220Z Gathering data for ns/openshift-multus... 2026-07-06T20:58:31.815368786Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-06T20:58:31.820058466Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T20:58:32.509331435Z Collecting kubelet logs for node ip-10-0-135-232.ec2.internal 2026-07-06T20:58:32.511279969Z Collecting kubelet logs for node ip-10-0-138-211.ec2.internal 2026-07-06T20:58:32.513069004Z Collecting kubelet logs for node ip-10-0-141-189.ec2.internal 2026-07-06T20:58:32.874276385Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-gwwrd namespace 2026-07-06T20:58:32.881049031Z INFO: Node performance data collection complete. 2026-07-06T20:58:33.229318333Z Gathering data for ns/openshift-host-network... 2026-07-06T20:58:33.638093515Z Gathering data for ns/openshift-network-diagnostics... 2026-07-06T20:58:34.196263203Z Gathering data for ns/openshift-network-node-identity... 2026-07-06T20:58:34.606326874Z Gathering data for ns/openshift-network-operator... 2026-07-06T20:58:34.978559540Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-06T20:58:35.245726305Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-06T20:58:35.808768796Z Gathering data for ns/openshift-apiserver-operator... 2026-07-06T20:58:36.157417314Z Gathering data for ns/openshift-apiserver... 2026-07-06T20:58:36.574916569Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-06T20:58:36.826030459Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T20:58:36.934100369Z Gathering data for ns/openshift-controller-manager... 2026-07-06T20:58:37.287503807Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-06T20:58:37.990524111Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-06T20:58:38.313314831Z Gathering data for ns/openshift-service-ca-operator... 2026-07-06T20:58:38.689185446Z Gathering data for ns/openshift-service-ca... 2026-07-06T20:58:39.150230567Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-06T20:58:40.205586207Z Wrote inspect data to must-gather. 2026-07-06T20:58:40.205626822Z error: inspection completed with the errors occurred while gathering data: 2026-07-06T20:58:40.205626822Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-06T20:58:40.330852165Z error: the server doesn't have a resource type "clusters" 2026-07-06T20:58:40.471736793Z Caches written to disk