2026-09-18T17:07:13.890498720Z [disk usage checker] Started 2026-09-18T17:07:13.894206678Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-18T17:07:14.103384848Z Gathering data for ns/openshift-cluster-version... 2026-09-18T17:07:14.253744374Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-18T17:07:14.351403780Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-18T17:07:14.440426694Z Gathering data for ns/default... 2026-09-18T17:07:14.753819533Z Gathering data for ns/openshift... 2026-09-18T17:07:15.186783849Z Gathering data for ns/kube-system... 2026-09-18T17:07:16.067903040Z Gathering data for ns/openshift-etcd... 2026-09-18T17:07:16.386766262Z Wrote inspect data to must-gather. 2026-09-18T17:07:16.386800484Z error: inspection completed with the errors occurred while gathering data: 2026-09-18T17:07:16.386800484Z namespaces "assisted-installer" not found 2026-09-18T17:07:16.730756349Z Waiting on subprocesses to finish execution. 2026-09-18T17:07:16.738832071Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-09-18T17:07:16.740981682Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-09-18T17:07:16.747044023Z INFO: Gathering HAProxy config files 2026-09-18T17:07:16.748658452Z INFO: Gathering machine config daemon's old logs from all nodes 2026-09-18T17:07:16.751639408Z INFO: Collecting host service logs for crio 2026-09-18T17:07:16.751925462Z INFO: Collecting host service logs for kubelet 2026-09-18T17:07:16.752248830Z INFO: Collecting host service logs for rpm-ostreed 2026-09-18T17:07:16.752618300Z INFO: Collecting host service logs for ostree-finalize-staged 2026-09-18T17:07:16.752939664Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-09-18T17:07:16.753247676Z INFO: Collecting host service logs for machine-config-daemon-host 2026-09-18T17:07:16.753566718Z INFO: Collecting host service logs for NetworkManager 2026-09-18T17:07:16.753911829Z INFO: Collecting host service logs for openvswitch 2026-09-18T17:07:16.754261233Z INFO: Collecting host service logs for ovs-configuration 2026-09-18T17:07:16.754648741Z INFO: Collecting host service logs for ovsdb-server 2026-09-18T17:07:16.754997764Z INFO: Collecting host service logs for ovs-vswitchd 2026-09-18T17:07:16.755426817Z INFO: Waiting for worker host service log collection to complete ... 2026-09-18T17:07:16.755568783Z Executing Istio gather script 2026-09-18T17:07:16.866269008Z INFO: Waiting for node performance related collection to complete ... 2026-09-18T17:07:17.477758332Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-09-18T17:07:17.477861749Z INFO: on-disk MachineConfig config collection complete. 2026-09-18T17:07:17.528295511Z INFO: Found 1 replicas - prometheus-k8s-0 2026-09-18T17:07:17.704969830Z INFO: Collecting Insights Archives from insights-operator-6f9c644dd9-6zxcq 2026-09-18T17:07:17.705015791Z insights-runtime-extractor-5zlr7 2026-09-18T17:07:17.705036046Z insights-runtime-extractor-f5szk 2026-09-18T17:07:17.705055606Z insights-runtime-extractor-n6t97 2026-09-18T17:07:17.776478473Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-09-18T17:07:17.836741303Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-09-18T17:07:17.854189529Z No resources found in openshift-etcd namespace. 2026-09-18T17:07:17.936628570Z No resources found 2026-09-18T17:07:18.091366091Z INFO: "metallb-operator" not detected. Skipping. 2026-09-18T17:07:18.219527455Z error: the server doesn't have a resource type "performanceprofile" 2026-09-18T17:07:18.353102173Z INFO: "sriov-network-operator" not detected. Skipping. 2026-09-18T17:07:18.398108282Z Wrote inspect data to must-gather. 2026-09-18T17:07:18.719543802Z Wrote inspect data to must-gather. 2026-09-18T17:07:18.722498707Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-09-18T17:07:18.722498707Z See 'oc rsync -h' for help and examples 2026-09-18T17:07:18.892283648Z Gathering data for ns/openshift-monitoring... 2026-09-18T17:07:18.923339543Z INFO: Worker host service log collection to complete. 2026-09-18T17:07:18.935096939Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-18T17:07:18.952327065Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-09-18T17:07:18.986091869Z INFO: Waiting for HAProxy config collection to complete ... 2026-09-18T17:07:19.079995475Z ERROR: No running kube-apiserver pods found 2026-09-18T17:07:19.155071429Z Wrote inspect data to must-gather. 2026-09-18T17:07:19.201598221Z INFO: 'previous-logs' folder not found on ip-10-0-128-224.ec2.internal, skipping... 2026-09-18T17:07:19.290130505Z Wrote inspect data to must-gather. 2026-09-18T17:07:19.329270504Z Wrote inspect data to must-gather. 2026-09-18T17:07:19.362364372Z Wrote inspect data to must-gather. 2026-09-18T17:07:19.561427907Z Wrote inspect data to must-gather. 2026-09-18T17:07:19.573367118Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-09-18T17:07:19.578810551Z No resources found 2026-09-18T17:07:19.668134053Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-18T17:07:19.768819725Z tar: Removing leading `/' from member names 2026-09-18T17:07:19.775270042Z INFO: HAProxy config collection complete. 2026-09-18T17:07:19.780721151Z error: the server doesn't have a resource type "machineconfigs" 2026-09-18T17:07:19.852457982Z INFO: 'previous-logs' folder not found on ip-10-0-135-110.ec2.internal, skipping... 2026-09-18T17:07:19.960910107Z Wrote inspect data to must-gather. 2026-09-18T17:07:20.037624681Z INFO: Getting rules from prometheus-k8s-0 2026-09-18T17:07:20.043913005Z error: the server doesn't have a resource type "machineconfigpools" 2026-09-18T17:07:20.191135090Z INFO: 'previous-logs' folder not found on ip-10-0-143-79.ec2.internal, skipping... 2026-09-18T17:07:20.191265700Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-09-18T17:07:20.191281273Z INFO: Machine Config Daemon termination log collection complete. 2026-09-18T17:07:20.328187295Z Wrote inspect data to must-gather. 2026-09-18T17:07:20.330674734Z Wrote inspect data to must-gather. 2026-09-18T17:07:20.466920170Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-09-18T17:07:20.509436480Z error: the server doesn't have a resource type "kubeletconfigs" 2026-09-18T17:07:20.531739452Z INFO: Getting status/config from prometheus-k8s-0 2026-09-18T17:07:20.605519533Z Wrote inspect data to must-gather. 2026-09-18T17:07:20.785885518Z Wrote inspect data to must-gather. 2026-09-18T17:07:20.793952863Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-18T17:07:20.949145799Z INFO: INTERCONNECT MODE 2026-09-18T17:07:20.949186262Z INFO: Gathering ovn-kubernetes DBs 2026-09-18T17:07:21.042151158Z INFO: Getting status/flags from prometheus-k8s-0 2026-09-18T17:07:21.049089705Z Wrote inspect data to must-gather. 2026-09-18T17:07:21.164780054Z INFO: Gathering OVN_Northbound from ovnkube-node-gw4fg... 2026-09-18T17:07:21.165033072Z INFO: Gathering OVN_Northbound from ovnkube-node-w9n9q... 2026-09-18T17:07:21.165346187Z INFO: Gathering OVN_Northbound from ovnkube-node-xmk47... 2026-09-18T17:07:21.165678194Z INFO: Gathering OVN_Southbound from ovnkube-node-gw4fg... 2026-09-18T17:07:21.165958265Z INFO: Gathering OVN_Southbound from ovnkube-node-w9n9q... 2026-09-18T17:07:21.166254258Z INFO: Gathering OVN_Southbound from ovnkube-node-xmk47... 2026-09-18T17:07:21.379498547Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-09-18T17:07:21.388990415Z Wrote inspect data to must-gather. 2026-09-18T17:07:21.472465693Z tar: Removing leading `/' from member names 2026-09-18T17:07:21.525518827Z tar: Removing leading `/' from member names 2026-09-18T17:07:21.526531615Z tar: Removing leading `/' from member names 2026-09-18T17:07:21.562761132Z tar: Removing leading `/' from member names 2026-09-18T17:07:21.591511846Z tar: Removing leading `/' from member names 2026-09-18T17:07:21.593008738Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-18T17:07:21.593503087Z tar: Removing leading `/' from member names 2026-09-18T17:07:21.723247929Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-09-18T17:07:21.805854054Z Wrote inspect data to must-gather. 2026-09-18T17:07:21.823757558Z Gathering data for ns/openshift-network-console... 2026-09-18T17:07:22.035554407Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-09-18T17:07:22.051457608Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-09-18T17:07:22.166983333Z INFO: Waiting for network log collection to complete ... 2026-09-18T17:07:22.167015300Z INFO: Waiting for ovnk database copies to complete ... 2026-09-18T17:07:22.167435958Z INFO: Copying ovnk databases complete. 2026-09-18T17:07:22.169108236Z 3.9M must-gather/network_logs/ovnk_database_store 2026-09-18T17:07:22.171155647Z ovnk_database_store/ 2026-09-18T17:07:22.171155647Z ovnk_database_store/ovnkube-node-w9n9q_nbdb 2026-09-18T17:07:22.173358190Z Wrote inspect data to must-gather. 2026-09-18T17:07:22.182635562Z ovnk_database_store/ovnkube-node-gw4fg_sbdb 2026-09-18T17:07:22.210402784Z ovnk_database_store/ovnkube-node-gw4fg_nbdb 2026-09-18T17:07:22.220082880Z ovnk_database_store/ovnkube-node-xmk47_sbdb 2026-09-18T17:07:22.247293652Z ovnk_database_store/ovnkube-node-xmk47_nbdb 2026-09-18T17:07:22.257397198Z ovnk_database_store/ovnkube-node-w9n9q_sbdb 2026-09-18T17:07:22.289146290Z INFO: Network log collection complete. 2026-09-18T17:07:22.301111182Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:22.431744989Z Gathering data for ns/openshift-console-operator... 2026-09-18T17:07:22.504891014Z INFO: Getting status from alertmanager-main-0 2026-09-18T17:07:22.553761850Z Wrote inspect data to must-gather. 2026-09-18T17:07:22.573595169Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:22.576114758Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-09-18T17:07:22.786649640Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:22.788618886Z W0918 17:07:22.788577 1544 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-09-18T17:07:22.884692687Z Wrote inspect data to must-gather. 2026-09-18T17:07:22.903758426Z Gathering data for ns/openshift-console... 2026-09-18T17:07:23.022792741Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:23.026006877Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-09-18T17:07:23.170451759Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-09-18T17:07:23.172519554Z INFO: Fallback to identify the container image from release info 2026-09-18T17:07:23.233614539Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:23.445193598Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:23.448197551Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-09-18T17:07:23.462549287Z Gathering data for ns/openshift-cluster-storage-operator... 2026-09-18T17:07:23.682657363Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:23.726575858Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0720c8b078cc0cebb01b4c5cde21180aee7c2bb1a0d322b21890c0d3707406ea 2026-09-18T17:07:23.879191362Z daemonset.apps/perf-node-gather-daemonset created 2026-09-18T17:07:23.906086208Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:23.909951865Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-09-18T17:07:24.003874965Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-18T17:07:24.041641593Z Waiting for performance profile collector pods to become ready: 1 2026-09-18T17:07:24.063061573Z Gathering data for ns/openshift-dns-operator... 2026-09-18T17:07:24.119624839Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:24.350454428Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:24.353419892Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-09-18T17:07:24.407480440Z Gathering data for ns/openshift-dns... 2026-09-18T17:07:24.525750591Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:24.724521645Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:24.727816300Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-09-18T17:07:24.897154334Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:25.093814495Z Gathering data for ns/openshift-image-registry... 2026-09-18T17:07:25.127208562Z Wrote inspect data to /must-gather/istio. 2026-09-18T17:07:25.186133803Z Waiting for performance profile collector pods to become ready: 2 2026-09-18T17:07:25.622979997Z Gathering data for ns/openshift-ingress-operator... 2026-09-18T17:07:25.843433185Z Done executing Istio gather script 2026-09-18T17:07:25.981004190Z Gathering data for ns/openshift-ingress... 2026-09-18T17:07:26.320890267Z Gathering data for ns/openshift-ingress-canary... 2026-09-18T17:07:26.322463723Z Waiting for performance profile collector pods to become ready: 3 2026-09-18T17:07:26.725337401Z Gathering data for ns/openshift-insights... 2026-09-18T17:07:27.448472024Z Waiting for performance profile collector pods to become ready: 4 2026-09-18T17:07:28.573793297Z Waiting for performance profile collector pods to become ready: 5 2026-09-18T17:07:29.008409942Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-18T17:07:29.135525104Z Gathering data for ns/kserve... 2026-09-18T17:07:29.700361424Z Waiting for performance profile collector pods to become ready: 6 2026-09-18T17:07:30.734399108Z Gathering data for ns/openshift-config... 2026-09-18T17:07:30.838520230Z Waiting for performance profile collector pods to become ready: 7 2026-09-18T17:07:31.003229462Z Gathering data for ns/openshift-config-managed... 2026-09-18T17:07:31.459639638Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-09-18T17:07:31.784034761Z Gathering data for ns/openshift-kube-apiserver... 2026-09-18T17:07:31.960965777Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-09-18T17:07:32.054208233Z Gathering data for ns/openshift-kube-controller-manager... 2026-09-18T17:07:32.241475516Z Collecting performance related data for node ip-10-0-143-79.ec2.internal 2026-09-18T17:07:32.248324598Z Collecting performance related data for node ip-10-0-135-110.ec2.internal 2026-09-18T17:07:32.258589275Z Collecting performance related data for node ip-10-0-128-224.ec2.internal 2026-09-18T17:07:32.394134412Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-09-18T17:07:32.732895649Z Gathering data for ns/openshift-kube-scheduler... 2026-09-18T17:07:33.044388355Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-09-18T17:07:33.437044089Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-09-18T17:07:33.857892312Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-09-18T17:07:34.012919031Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-18T17:07:34.179226784Z Gathering data for ns/openshift-user-workload-monitoring... 2026-09-18T17:07:34.770437980Z Gathering data for ns/openshift-multus... 2026-09-18T17:07:36.183030448Z Gathering data for ns/openshift-ovn-kubernetes... 2026-09-18T17:07:37.501869701Z Gathering data for ns/openshift-host-network... 2026-09-18T17:07:37.982212204Z Gathering data for ns/openshift-network-diagnostics... 2026-09-18T17:07:38.578177069Z Gathering data for ns/openshift-network-node-identity... 2026-09-18T17:07:39.019724463Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-18T17:07:39.026518247Z Gathering data for ns/openshift-network-operator... 2026-09-18T17:07:39.436802069Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-09-18T17:07:39.729839743Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-09-18T17:07:40.225223488Z Gathering data for ns/openshift-apiserver-operator... 2026-09-18T17:07:40.510386115Z Gathering data for ns/openshift-apiserver... 2026-09-18T17:07:40.624539979Z Collecting kubelet logs for node ip-10-0-143-79.ec2.internal 2026-09-18T17:07:40.626294199Z Collecting kubelet logs for node ip-10-0-135-110.ec2.internal 2026-09-18T17:07:40.628285475Z Collecting kubelet logs for node ip-10-0-128-224.ec2.internal 2026-09-18T17:07:40.975732615Z Gathering data for ns/openshift-controller-manager-operator... 2026-09-18T17:07:41.070595203Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-zpw2m namespace 2026-09-18T17:07:41.076239170Z INFO: Node performance data collection complete. 2026-09-18T17:07:41.268030196Z Gathering data for ns/openshift-controller-manager... 2026-09-18T17:07:41.584559023Z Gathering data for ns/openshift-cluster-samples-operator... 2026-09-18T17:07:42.219848390Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-09-18T17:07:42.536628846Z Gathering data for ns/openshift-service-ca-operator... 2026-09-18T17:07:42.871066498Z Gathering data for ns/openshift-service-ca... 2026-09-18T17:07:43.246371577Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-09-18T17:07:44.024584993Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-09-18T17:07:44.159755819Z Wrote inspect data to must-gather. 2026-09-18T17:07:44.159994262Z error: inspection completed with the errors occurred while gathering data: 2026-09-18T17:07:44.159994262Z skipping gathering secrets/support due to error: secrets "support" not found 2026-09-18T17:07:44.289277100Z error: the server doesn't have a resource type "clusters" 2026-09-18T17:07:44.445201292Z Caches written to disk