2026-07-01T19:50:25.568929773Z [disk usage checker] Started 2026-07-01T19:50:25.572565763Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 2026-07-01T19:50:25.776845288Z Gathering data for ns/openshift-cluster-version... 2026-07-01T19:50:25.924910166Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-01T19:50:26.026831010Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-01T19:50:26.121086038Z Gathering data for ns/default... 2026-07-01T19:50:26.472626977Z Gathering data for ns/openshift... 2026-07-01T19:50:26.928992057Z Gathering data for ns/kube-system... 2026-07-01T19:50:27.878692456Z Gathering data for ns/openshift-etcd... 2026-07-01T19:50:28.226169687Z Wrote inspect data to must-gather. 2026-07-01T19:50:28.226202981Z error: inspection completed with the errors occurred while gathering data: 2026-07-01T19:50:28.226202981Z namespaces "assisted-installer" not found 2026-07-01T19:50:28.474087820Z Waiting on subprocesses to finish execution. 2026-07-01T19:50:28.481619442Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-01T19:50:28.493333328Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-01T19:50:28.493477001Z INFO: Gathering HAProxy config files 2026-07-01T19:50:28.493653493Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-01T19:50:28.495647645Z INFO: Collecting host service logs for crio 2026-07-01T19:50:28.495965452Z INFO: Collecting host service logs for kubelet 2026-07-01T19:50:28.496281746Z INFO: Collecting host service logs for rpm-ostreed 2026-07-01T19:50:28.496612856Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-01T19:50:28.496648184Z Executing Istio gather script 2026-07-01T19:50:28.496967946Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-01T19:50:28.497302536Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-01T19:50:28.497643154Z INFO: Collecting host service logs for NetworkManager 2026-07-01T19:50:28.498839889Z INFO: Collecting host service logs for openvswitch 2026-07-01T19:50:28.499273221Z INFO: Collecting host service logs for ovs-configuration 2026-07-01T19:50:28.499678202Z INFO: Collecting host service logs for ovsdb-server 2026-07-01T19:50:28.500010216Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-01T19:50:28.500387960Z INFO: Waiting for worker host service log collection to complete ... 2026-07-01T19:50:28.525614843Z INFO: Waiting for node performance related collection to complete ... 2026-07-01T19:50:28.955712537Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-01T19:50:29.201684427Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-01T19:50:29.201772589Z INFO: on-disk MachineConfig config collection complete. 2026-07-01T19:50:29.338307630Z No resources found 2026-07-01T19:50:29.400802738Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-gmpj2 2026-07-01T19:50:29.400802738Z insights-runtime-extractor-9x8mw 2026-07-01T19:50:29.400802738Z insights-runtime-extractor-kq5nv 2026-07-01T19:50:29.400802738Z insights-runtime-extractor-sxxm4 2026-07-01T19:50:29.529816607Z Wrote inspect data to must-gather. 2026-07-01T19:50:29.603367245Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-01T19:50:29.612420624Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-01T19:50:29.667174362Z No resources found in openshift-etcd namespace. 2026-07-01T19:50:29.712009555Z INFO: "metallb-operator" not detected. Skipping. 2026-07-01T19:50:29.776102559Z ERROR: No running kube-apiserver pods found 2026-07-01T19:50:29.927775402Z error: the server doesn't have a resource type "performanceprofile" 2026-07-01T19:50:30.021734349Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-01T19:50:30.042965342Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-01T19:50:30.078007530Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-01T19:50:30.078007530Z See 'oc rsync -h' for help and examples 2026-07-01T19:50:30.174180479Z INFO: Worker host service log collection to complete. 2026-07-01T19:50:30.402610166Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-01T19:50:30.586761045Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 2026-07-01T19:50:30.674250783Z Wrote inspect data to must-gather. 2026-07-01T19:50:30.704092594Z No resources found 2026-07-01T19:50:30.731751327Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-01T19:50:30.847441737Z tar: Removing leading `/' from member names 2026-07-01T19:50:30.851454955Z INFO: HAProxy config collection complete. 2026-07-01T19:50:30.916812283Z Wrote inspect data to must-gather. 2026-07-01T19:50:30.978246262Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-01T19:50:31.059936018Z Wrote inspect data to must-gather. 2026-07-01T19:50:31.073451655Z Wrote inspect data to must-gather. 2026-07-01T19:50:31.152875947Z Wrote inspect data to must-gather. 2026-07-01T19:50:31.169738405Z Gathering data for ns/openshift-monitoring... 2026-07-01T19:50:31.212677056Z Wrote inspect data to must-gather. 2026-07-01T19:50:31.274252013Z INFO: 'previous-logs' folder not found on ip-10-0-132-119.ec2.internal, skipping... 2026-07-01T19:50:31.411201233Z Wrote inspect data to must-gather. 2026-07-01T19:50:31.440124324Z error: the server doesn't have a resource type "machineconfigs" 2026-07-01T19:50:31.501216437Z INFO: Getting rules from prometheus-k8s-0 2026-07-01T19:50:31.600823048Z INFO: 'previous-logs' folder not found on ip-10-0-138-218.ec2.internal, skipping... 2026-07-01T19:50:31.658723758Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-01T19:50:31.746859940Z Wrote inspect data to must-gather. 2026-07-01T19:50:31.918960078Z INFO: 'previous-logs' folder not found on ip-10-0-140-188.ec2.internal, skipping... 2026-07-01T19:50:31.919069361Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-01T19:50:31.919119377Z INFO: Machine Config Daemon termination log collection complete. 2026-07-01T19:50:31.955253895Z Wrote inspect data to must-gather. 2026-07-01T19:50:32.024016777Z INFO: Getting status/config from prometheus-k8s-0 2026-07-01T19:50:32.044501754Z Wrote inspect data to must-gather. 2026-07-01T19:50:32.150390740Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-01T19:50:32.320592452Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-01T19:50:32.372940603Z INFO: INTERCONNECT MODE 2026-07-01T19:50:32.373030124Z INFO: Gathering ovn-kubernetes DBs 2026-07-01T19:50:32.403960415Z Wrote inspect data to must-gather. 2026-07-01T19:50:32.492444591Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-01T19:50:32.545525161Z INFO: Gathering OVN_Northbound from ovnkube-node-92vsf... 2026-07-01T19:50:32.545751445Z INFO: Gathering OVN_Northbound from ovnkube-node-f8jjd... 2026-07-01T19:50:32.545965502Z INFO: Gathering OVN_Northbound from ovnkube-node-mhs48... 2026-07-01T19:50:32.546265381Z INFO: Gathering OVN_Southbound from ovnkube-node-92vsf... 2026-07-01T19:50:32.546510365Z INFO: Gathering OVN_Southbound from ovnkube-node-f8jjd... 2026-07-01T19:50:32.546844294Z INFO: Gathering OVN_Southbound from ovnkube-node-mhs48... 2026-07-01T19:50:32.741451257Z Wrote inspect data to must-gather. 2026-07-01T19:50:32.854994945Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-01T19:50:32.879676741Z tar: Removing leading `/' from member names 2026-07-01T19:50:32.916637144Z tar: Removing leading `/' from member names 2026-07-01T19:50:32.916738363Z tar: Removing leading `/' from member names 2026-07-01T19:50:32.920488848Z tar: Removing leading `/' from member names 2026-07-01T19:50:32.931078163Z tar: Removing leading `/' from member names 2026-07-01T19:50:32.931466760Z tar: Removing leading `/' from member names 2026-07-01T19:50:33.061867150Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-01T19:50:33.065030569Z Wrote inspect data to must-gather. 2026-07-01T19:50:33.150956525Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-01T19:50:33.331982402Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-01T19:50:33.360625875Z Wrote inspect data to must-gather. 2026-07-01T19:50:33.447357313Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-01T19:50:33.515551495Z INFO: Waiting for network log collection to complete ... 2026-07-01T19:50:33.515607923Z INFO: Waiting for ovnk database copies to complete ... 2026-07-01T19:50:33.516095884Z INFO: Copying ovnk databases complete. 2026-07-01T19:50:33.517770946Z 3.4M must-gather/network_logs/ovnk_database_store 2026-07-01T19:50:33.520125643Z ovnk_database_store/ 2026-07-01T19:50:33.520173631Z ovnk_database_store/ovnkube-node-mhs48_nbdb 2026-07-01T19:50:33.529431076Z ovnk_database_store/ovnkube-node-mhs48_sbdb 2026-07-01T19:50:33.553853536Z ovnk_database_store/ovnkube-node-92vsf_nbdb 2026-07-01T19:50:33.562909177Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:33.563028808Z ovnk_database_store/ovnkube-node-92vsf_sbdb 2026-07-01T19:50:33.587824631Z ovnk_database_store/ovnkube-node-f8jjd_sbdb 2026-07-01T19:50:33.607986351Z ovnk_database_store/ovnkube-node-f8jjd_nbdb 2026-07-01T19:50:33.608530657Z Wrote inspect data to must-gather. 2026-07-01T19:50:33.620329422Z INFO: Network log collection complete. 2026-07-01T19:50:33.779026293Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-01T19:50:33.811474407Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:33.814751403Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-01T19:50:33.903105452Z INFO: Getting status from alertmanager-main-0 2026-07-01T19:50:33.931163009Z Wrote inspect data to must-gather. 2026-07-01T19:50:33.978139846Z Gathering data for ns/openshift-network-console... 2026-07-01T19:50:34.043466237Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:34.254724224Z Wrote inspect data to must-gather. 2026-07-01T19:50:34.278157490Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:34.280886162Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-01T19:50:34.460013940Z Gathering data for ns/openshift-console-operator... 2026-07-01T19:50:34.495475809Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:34.569769618Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-01T19:50:34.572526329Z INFO: Fallback to identify the container image from release info 2026-07-01T19:50:34.712626718Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:34.715879102Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-01T19:50:34.849664300Z Gathering data for ns/openshift-console... 2026-07-01T19:50:34.940144893Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:35.143720207Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:35.148838774Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-01T19:50:35.158310075Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f 2026-07-01T19:50:35.306676586Z daemonset.apps/perf-node-gather-daemonset created 2026-07-01T19:50:35.348169002Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:35.405315400Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-01T19:50:35.474686998Z Waiting for performance profile collector pods to become ready: 1 2026-07-01T19:50:35.576368874Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:35.578686661Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-01T19:50:35.618444502Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 2026-07-01T19:50:35.784489281Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:35.976325480Z Gathering data for ns/openshift-dns-operator... 2026-07-01T19:50:36.021779255Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:36.025093528Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-01T19:50:36.211096280Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:36.263061367Z Gathering data for ns/openshift-dns... 2026-07-01T19:50:36.399820324Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:36.402094987Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-01T19:50:36.592232223Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:36.621384181Z Waiting for performance profile collector pods to become ready: 2 2026-07-01T19:50:36.802877899Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:36.805714177Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-01T19:50:36.891984502Z Gathering data for ns/openshift-image-registry... 2026-07-01T19:50:36.979370662Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:37.180551015Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:37.185877060Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-01T19:50:37.371449753Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:37.384222789Z Gathering data for ns/openshift-ingress-operator... 2026-07-01T19:50:37.586517576Z Wrote inspect data to /must-gather/istio. 2026-07-01T19:50:37.751903017Z Gathering data for ns/openshift-ingress... 2026-07-01T19:50:37.771645036Z Waiting for performance profile collector pods to become ready: 3 2026-07-01T19:50:38.093346409Z Gathering data for ns/openshift-ingress-canary... 2026-07-01T19:50:38.328937639Z Done executing Istio gather script 2026-07-01T19:50:38.484529883Z Gathering data for ns/openshift-insights... 2026-07-01T19:50:38.894158633Z Waiting for performance profile collector pods to become ready: 4 2026-07-01T19:50:40.036404928Z Waiting for performance profile collector pods to become ready: 5 2026-07-01T19:50:40.623373863Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 2026-07-01T19:50:40.703318840Z Gathering data for ns/kserve... 2026-07-01T19:50:41.155239758Z Waiting for performance profile collector pods to become ready: 6 2026-07-01T19:50:42.293874298Z Waiting for performance profile collector pods to become ready: 7 2026-07-01T19:50:42.352854828Z Gathering data for ns/openshift-config... 2026-07-01T19:50:42.606538710Z Gathering data for ns/openshift-config-managed... 2026-07-01T19:50:43.031982098Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-01T19:50:43.287992697Z Gathering data for ns/openshift-kube-apiserver... 2026-07-01T19:50:43.429198739Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-01T19:50:43.572313454Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-01T19:50:43.712457885Z Collecting performance related data for node ip-10-0-138-218.ec2.internal 2026-07-01T19:50:43.719634545Z Collecting performance related data for node ip-10-0-140-188.ec2.internal 2026-07-01T19:50:43.727803640Z Collecting performance related data for node ip-10-0-132-119.ec2.internal 2026-07-01T19:50:43.933534818Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-01T19:50:44.332902289Z Gathering data for ns/openshift-kube-scheduler... 2026-07-01T19:50:44.720764523Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-01T19:50:45.103945443Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-01T19:50:45.517539536Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-01T19:50:45.628028318Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 2026-07-01T19:50:45.821261996Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-01T19:50:46.339609345Z Gathering data for ns/openshift-multus... 2026-07-01T19:50:47.979120744Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-01T19:50:49.218041872Z Gathering data for ns/openshift-host-network... 2026-07-01T19:50:49.525035865Z Gathering data for ns/openshift-network-diagnostics... 2026-07-01T19:50:50.059475351Z Gathering data for ns/openshift-network-node-identity... 2026-07-01T19:50:50.453020418Z Gathering data for ns/openshift-network-operator... 2026-07-01T19:50:50.632598935Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 2026-07-01T19:50:50.813807540Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-01T19:50:51.091998207Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-01T19:50:51.576803914Z Gathering data for ns/openshift-apiserver-operator... 2026-07-01T19:50:51.867130932Z Gathering data for ns/openshift-apiserver... 2026-07-01T19:50:52.244046366Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-01T19:50:52.382620670Z Collecting kubelet logs for node ip-10-0-138-218.ec2.internal 2026-07-01T19:50:52.384352354Z Collecting kubelet logs for node ip-10-0-140-188.ec2.internal 2026-07-01T19:50:52.386023420Z Collecting kubelet logs for node ip-10-0-132-119.ec2.internal 2026-07-01T19:50:52.619304467Z Gathering data for ns/openshift-controller-manager... 2026-07-01T19:50:52.788293681Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-qhllw namespace 2026-07-01T19:50:52.793662409Z INFO: Node performance data collection complete. 2026-07-01T19:50:52.899458855Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-01T19:50:53.550316488Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-01T19:50:53.877366715Z Gathering data for ns/openshift-service-ca-operator... 2026-07-01T19:50:54.208387916Z Gathering data for ns/openshift-service-ca... 2026-07-01T19:50:54.576881444Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-01T19:50:55.531292271Z Wrote inspect data to must-gather. 2026-07-01T19:50:55.531342018Z error: inspection completed with the errors occurred while gathering data: 2026-07-01T19:50:55.531342018Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-01T19:50:55.637079494Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70 2026-07-01T19:50:55.667674976Z error: the server doesn't have a resource type "clusters" 2026-07-01T19:50:55.823375920Z Caches written to disk