2026-09-08T14:08:41.963141875Z [disk usage checker] Started 2026-09-08T14:08:41.966733504Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-08T14:08:42.157352858Z Gathering data for ns/openshift-cluster-version... 2026-09-08T14:08:42.296063816Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-08T14:08:42.389311066Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-08T14:08:42.469828792Z Gathering data for ns/default... 2026-09-08T14:08:42.788294314Z Gathering data for ns/openshift... 2026-09-08T14:08:43.207870264Z Gathering data for ns/kube-system... 2026-09-08T14:08:44.042069309Z Gathering data for ns/openshift-etcd... 2026-09-08T14:08:44.383452749Z Wrote inspect data to must-gather. 2026-09-08T14:08:44.383549819Z error: inspection completed with the errors occurred while gathering data: 2026-09-08T14:08:44.383549819Z namespaces "assisted-installer" not found 2026-09-08T14:08:44.836279324Z Waiting on subprocesses to finish execution. 2026-09-08T14:08:44.842623177Z Executing Istio gather script 2026-09-08T14:08:44.847346236Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-09-08T14:08:44.849603506Z INFO: Gathering HAProxy config files 2026-09-08T14:08:44.857853293Z INFO: Collecting host service logs for crio 2026-09-08T14:08:44.858223714Z INFO: Collecting host service logs for kubelet 2026-09-08T14:08:44.858606115Z INFO: Collecting host service logs for rpm-ostreed 2026-09-08T14:08:44.858946369Z INFO: Collecting host service logs for ostree-finalize-staged 2026-09-08T14:08:44.859272992Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-09-08T14:08:44.859704484Z INFO: Collecting host service logs for machine-config-daemon-host 2026-09-08T14:08:44.860046941Z INFO: Collecting host service logs for NetworkManager 2026-09-08T14:08:44.860364437Z INFO: Collecting host service logs for openvswitch 2026-09-08T14:08:44.860706751Z INFO: Collecting host service logs for ovs-configuration 2026-09-08T14:08:44.861060971Z INFO: Collecting host service logs for ovsdb-server 2026-09-08T14:08:44.861433734Z INFO: Collecting host service logs for ovs-vswitchd 2026-09-08T14:08:44.861974574Z INFO: Waiting for worker host service log collection to complete ... 2026-09-08T14:08:44.864648381Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-09-08T14:08:44.883117408Z INFO: Gathering machine config daemon's old logs from all nodes 2026-09-08T14:08:44.889818439Z INFO: Waiting for node performance related collection to complete ... 2026-09-08T14:08:45.703504806Z No resources found 2026-09-08T14:08:45.723589025Z INFO: "sriov-network-operator" not detected. Skipping. 2026-09-08T14:08:45.751919063Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-09-08T14:08:45.758348679Z No resources found in openshift-etcd namespace. 2026-09-08T14:08:45.942421934Z Wrote inspect data to must-gather. 2026-09-08T14:08:45.965446923Z INFO: "metallb-operator" not detected. Skipping. 2026-09-08T14:08:46.006386099Z error: the server doesn't have a resource type "performanceprofile" 2026-09-08T14:08:46.028545359Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-kjmxs 2026-09-08T14:08:46.028590524Z insights-runtime-extractor-bdzv9 2026-09-08T14:08:46.028611285Z insights-runtime-extractor-nrnws 2026-09-08T14:08:46.028630693Z insights-runtime-extractor-vvcsd 2026-09-08T14:08:46.259893041Z INFO: Found 1 replicas - prometheus-k8s-0 2026-09-08T14:08:46.315798868Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-09-08T14:08:46.315798868Z See 'oc rsync -h' for help and examples 2026-09-08T14:08:46.441243179Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-09-08T14:08:46.441324015Z INFO: on-disk MachineConfig config collection complete. 2026-09-08T14:08:46.447959039Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-09-08T14:08:46.490328732Z INFO: Worker host service log collection to complete. 2026-09-08T14:08:46.574079289Z Wrote inspect data to must-gather. 2026-09-08T14:08:46.937612915Z Wrote inspect data to must-gather. 2026-09-08T14:08:46.945266493Z ERROR: No running kube-apiserver pods found 2026-09-08T14:08:46.985117611Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-08T14:08:47.145195532Z INFO: Waiting for HAProxy config collection to complete ... 2026-09-08T14:08:47.151039438Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-09-08T14:08:47.158277513Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-08T14:08:47.283545578Z Wrote inspect data to must-gather. 2026-09-08T14:08:47.303969275Z No resources found 2026-09-08T14:08:47.315637234Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-09-08T14:08:47.336996316Z Wrote inspect data to must-gather. 2026-09-08T14:08:47.438976537Z Wrote inspect data to must-gather. 2026-09-08T14:08:47.511714227Z INFO: 'previous-logs' folder not found on ip-10-0-133-71.ec2.internal, skipping... 2026-09-08T14:08:47.518690701Z Gathering data for ns/openshift-monitoring... 2026-09-08T14:08:47.614021324Z error: the server doesn't have a resource type "machineconfigs" 2026-09-08T14:08:47.715104285Z Wrote inspect data to must-gather. 2026-09-08T14:08:47.811273158Z tar: Removing leading `/' from member names 2026-09-08T14:08:47.819172628Z INFO: HAProxy config collection complete. 2026-09-08T14:08:47.850877960Z Wrote inspect data to must-gather. 2026-09-08T14:08:47.885549391Z error: the server doesn't have a resource type "machineconfigpools" 2026-09-08T14:08:47.939525736Z INFO: 'previous-logs' folder not found on ip-10-0-140-239.ec2.internal, skipping... 2026-09-08T14:08:48.048227861Z INFO: Getting rules from prometheus-k8s-0 2026-09-08T14:08:48.171674871Z Wrote inspect data to must-gather. 2026-09-08T14:08:48.204542007Z Wrote inspect data to must-gather. 2026-09-08T14:08:48.273004796Z INFO: 'previous-logs' folder not found on ip-10-0-141-177.ec2.internal, skipping... 2026-09-08T14:08:48.273041901Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-09-08T14:08:48.273049255Z INFO: Machine Config Daemon termination log collection complete. 2026-09-08T14:08:48.371537118Z error: the server doesn't have a resource type "kubeletconfigs" 2026-09-08T14:08:48.474085548Z Wrote inspect data to must-gather. 2026-09-08T14:08:48.602015980Z INFO: Getting status/config from prometheus-k8s-0 2026-09-08T14:08:48.642473642Z Wrote inspect data to must-gather. 2026-09-08T14:08:48.852119713Z INFO: INTERCONNECT MODE 2026-09-08T14:08:48.852167256Z INFO: Gathering ovn-kubernetes DBs 2026-09-08T14:08:48.892076532Z Wrote inspect data to must-gather. 2026-09-08T14:08:48.953070237Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-09-08T14:08:49.051284495Z INFO: Gathering OVN_Northbound from ovnkube-node-bd6fv... 2026-09-08T14:08:49.052279633Z INFO: Gathering OVN_Northbound from ovnkube-node-bfnz6... 2026-09-08T14:08:49.052513817Z INFO: Gathering OVN_Northbound from ovnkube-node-xxmjp... 2026-09-08T14:08:49.052791647Z INFO: Gathering OVN_Southbound from ovnkube-node-bd6fv... 2026-09-08T14:08:49.053088115Z INFO: Gathering OVN_Southbound from ovnkube-node-bfnz6... 2026-09-08T14:08:49.053371899Z INFO: Gathering OVN_Southbound from ovnkube-node-xxmjp... 2026-09-08T14:08:49.157021333Z INFO: Getting status/flags from prometheus-k8s-0 2026-09-08T14:08:49.205219172Z Wrote inspect data to must-gather. 2026-09-08T14:08:49.379589537Z tar: Removing leading `/' from member names 2026-09-08T14:08:49.396710414Z tar: Removing leading `/' from member names 2026-09-08T14:08:49.455314850Z tar: Removing leading `/' from member names 2026-09-08T14:08:49.456108185Z tar: Removing leading `/' from member names 2026-09-08T14:08:49.460446733Z tar: Removing leading `/' from member names 2026-09-08T14:08:49.474540845Z tar: Removing leading `/' from member names 2026-09-08T14:08:49.562049033Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-08T14:08:49.592972680Z Wrote inspect data to must-gather. 2026-09-08T14:08:49.615026600Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-09-08T14:08:49.883472387Z Wrote inspect data to must-gather. 2026-09-08T14:08:49.911166959Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-09-08T14:08:50.018010659Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-09-08T14:08:50.048874978Z INFO: Waiting for network log collection to complete ... 2026-09-08T14:08:50.048921468Z INFO: Waiting for ovnk database copies to complete ... 2026-09-08T14:08:50.049487998Z INFO: Copying ovnk databases complete. 2026-09-08T14:08:50.052245698Z 3.9M must-gather/network_logs/ovnk_database_store 2026-09-08T14:08:50.055754087Z ovnk_database_store/ 2026-09-08T14:08:50.055929854Z ovnk_database_store/ovnkube-node-bd6fv_sbdb 2026-09-08T14:08:50.087551981Z ovnk_database_store/ovnkube-node-bd6fv_nbdb 2026-09-08T14:08:50.097985482Z ovnk_database_store/ovnkube-node-xxmjp_sbdb 2026-09-08T14:08:50.122186523Z ovnk_database_store/ovnkube-node-xxmjp_nbdb 2026-09-08T14:08:50.131593610Z ovnk_database_store/ovnkube-node-bfnz6_sbdb 2026-09-08T14:08:50.156756270Z ovnk_database_store/ovnkube-node-bfnz6_nbdb 2026-09-08T14:08:50.158230846Z W0908 14:08:50.158171 1456 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-09-08T14:08:50.163679025Z Wrote inspect data to must-gather. 2026-09-08T14:08:50.170651425Z INFO: Network log collection complete. 2026-09-08T14:08:50.218724851Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-09-08T14:08:50.241805481Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:50.300221510Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-08T14:08:50.490680432Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:50.493786233Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-09-08T14:08:50.502844469Z Wrote inspect data to must-gather. 2026-09-08T14:08:50.512147960Z Gathering data for ns/openshift-network-console... 2026-09-08T14:08:50.685951847Z INFO: Getting status from alertmanager-main-0 2026-09-08T14:08:50.721171692Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:50.833759245Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-09-08T14:08:50.837233352Z INFO: Fallback to identify the container image from release info 2026-09-08T14:08:50.960886437Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:50.963348976Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-09-08T14:08:51.113475172Z Gathering data for ns/openshift-console-operator... 2026-09-08T14:08:51.169567013Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:51.363325612Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:51.365622305Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-09-08T14:08:51.499740902Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399 2026-09-08T14:08:51.553927112Z Gathering data for ns/openshift-console... 2026-09-08T14:08:51.631246241Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:51.660207972Z daemonset.apps/perf-node-gather-daemonset created 2026-09-08T14:08:51.812219678Z Waiting for performance profile collector pods to become ready: 1 2026-09-08T14:08:51.858986306Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:51.861955382Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-09-08T14:08:51.991356410Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-08T14:08:52.077669037Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:52.188581810Z Gathering data for ns/openshift-cluster-storage-operator... 2026-09-08T14:08:52.302049177Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:52.304556680Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-09-08T14:08:52.476211368Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:52.702483880Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:52.705127692Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-09-08T14:08:52.744827784Z Gathering data for ns/openshift-dns-operator... 2026-09-08T14:08:52.889927675Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:52.939563483Z Waiting for performance profile collector pods to become ready: 2 2026-09-08T14:08:53.103436881Z Wrote inspect data to /must-gather/istio. 2026-09-08T14:08:53.107788095Z Gathering data for ns/openshift-dns... 2026-09-08T14:08:53.755451690Z Gathering data for ns/openshift-image-registry... 2026-09-08T14:08:53.813470814Z Done executing Istio gather script 2026-09-08T14:08:54.057706547Z Waiting for performance profile collector pods to become ready: 3 2026-09-08T14:08:54.248511501Z Gathering data for ns/openshift-ingress-operator... 2026-09-08T14:08:54.615051337Z Gathering data for ns/openshift-ingress... 2026-09-08T14:08:54.979164658Z Gathering data for ns/openshift-ingress-canary... 2026-09-08T14:08:55.197893105Z Waiting for performance profile collector pods to become ready: 4 2026-09-08T14:08:55.411914993Z Gathering data for ns/openshift-insights... 2026-09-08T14:08:56.319516750Z Waiting for performance profile collector pods to become ready: 5 2026-09-08T14:08:56.997201706Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-08T14:08:57.441262644Z Waiting for performance profile collector pods to become ready: 6 2026-09-08T14:08:57.894429191Z Gathering data for ns/kserve... 2026-09-08T14:08:58.585180853Z Waiting for performance profile collector pods to become ready: 7 2026-09-08T14:08:59.565215096Z Gathering data for ns/openshift-config... 2026-09-08T14:08:59.714979355Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-09-08T14:08:59.866885909Z Gathering data for ns/openshift-config-managed... 2026-09-08T14:09:00.006269680Z Collecting performance related data for node ip-10-0-140-239.ec2.internal 2026-09-08T14:09:00.013165518Z Collecting performance related data for node ip-10-0-133-71.ec2.internal 2026-09-08T14:09:00.022374444Z Collecting performance related data for node ip-10-0-141-177.ec2.internal 2026-09-08T14:09:00.364759189Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-09-08T14:09:00.719119593Z Gathering data for ns/openshift-kube-apiserver... 2026-09-08T14:09:01.094137988Z Gathering data for ns/openshift-kube-controller-manager... 2026-09-08T14:09:01.461562544Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-09-08T14:09:01.827019252Z Gathering data for ns/openshift-kube-scheduler... 2026-09-08T14:09:02.003962588Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-08T14:09:02.136085783Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-09-08T14:09:02.454733833Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-09-08T14:09:02.852459895Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-09-08T14:09:03.231863985Z Gathering data for ns/openshift-user-workload-monitoring... 2026-09-08T14:09:03.800747734Z Gathering data for ns/openshift-multus... 2026-09-08T14:09:05.353491551Z Gathering data for ns/openshift-ovn-kubernetes... 2026-09-08T14:09:06.653020639Z Gathering data for ns/openshift-host-network... 2026-09-08T14:09:07.009765888Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-08T14:09:07.022770124Z Gathering data for ns/openshift-network-diagnostics... 2026-09-08T14:09:07.610869103Z Gathering data for ns/openshift-network-node-identity... 2026-09-08T14:09:08.040778769Z Gathering data for ns/openshift-network-operator... 2026-09-08T14:09:08.410881473Z Collecting kubelet logs for node ip-10-0-140-239.ec2.internal 2026-09-08T14:09:08.412795765Z Collecting kubelet logs for node ip-10-0-133-71.ec2.internal 2026-09-08T14:09:08.414493033Z Collecting kubelet logs for node ip-10-0-141-177.ec2.internal 2026-09-08T14:09:08.435767443Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-09-08T14:09:08.808128560Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-09-08T14:09:08.829881730Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-p58cr namespace 2026-09-08T14:09:08.834537687Z INFO: Node performance data collection complete. 2026-09-08T14:09:09.335872788Z Gathering data for ns/openshift-apiserver-operator... 2026-09-08T14:09:09.688655733Z Gathering data for ns/openshift-apiserver... 2026-09-08T14:09:10.144423311Z Gathering data for ns/openshift-controller-manager-operator... 2026-09-08T14:09:10.471021268Z Gathering data for ns/openshift-controller-manager... 2026-09-08T14:09:10.801241194Z Gathering data for ns/openshift-cluster-samples-operator... 2026-09-08T14:09:11.477815157Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-09-08T14:09:11.808721426Z Gathering data for ns/openshift-service-ca-operator... 2026-09-08T14:09:12.015925395Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-08T14:09:12.128794905Z Gathering data for ns/openshift-service-ca... 2026-09-08T14:09:12.539196056Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-09-08T14:09:13.474956311Z Wrote inspect data to must-gather. 2026-09-08T14:09:13.475000741Z error: inspection completed with the errors occurred while gathering data: 2026-09-08T14:09:13.475000741Z skipping gathering secrets/support due to error: secrets "support" not found 2026-09-08T14:09:13.608233110Z error: the server doesn't have a resource type "clusters" 2026-09-08T14:09:13.763737062Z Caches written to disk