2026-07-08T16:57:54.721522699Z [disk usage checker] Started 2026-07-08T16:57:54.725025003Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-08T16:57:54.939697934Z Gathering data for ns/openshift-cluster-version... 2026-07-08T16:57:55.084779738Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-08T16:57:55.179068108Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-08T16:57:55.260962886Z Gathering data for ns/default... 2026-07-08T16:57:55.582562983Z Gathering data for ns/openshift... 2026-07-08T16:57:55.990254867Z Gathering data for ns/kube-system... 2026-07-08T16:57:56.709872584Z Gathering data for ns/openshift-etcd... 2026-07-08T16:57:57.056227915Z Wrote inspect data to must-gather. 2026-07-08T16:57:57.056264694Z error: inspection completed with the errors occurred while gathering data: 2026-07-08T16:57:57.056264694Z namespaces "assisted-installer" not found 2026-07-08T16:57:57.545265446Z Waiting on subprocesses to finish execution. 2026-07-08T16:57:57.559721784Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-08T16:57:57.562816097Z Executing Istio gather script 2026-07-08T16:57:57.564910664Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-08T16:57:57.567584848Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-08T16:57:57.576660533Z INFO: Collecting host service logs for crio 2026-07-08T16:57:57.577060697Z INFO: Collecting host service logs for kubelet 2026-07-08T16:57:57.577447695Z INFO: Collecting host service logs for rpm-ostreed 2026-07-08T16:57:57.577826067Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-08T16:57:57.578221475Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-08T16:57:57.578607693Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-08T16:57:57.578995656Z INFO: Collecting host service logs for NetworkManager 2026-07-08T16:57:57.579398729Z INFO: Collecting host service logs for openvswitch 2026-07-08T16:57:57.579745621Z INFO: Collecting host service logs for ovs-configuration 2026-07-08T16:57:57.580105738Z INFO: Collecting host service logs for ovsdb-server 2026-07-08T16:57:57.580469321Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-08T16:57:57.580865631Z INFO: Waiting for worker host service log collection to complete ... 2026-07-08T16:57:57.584703340Z INFO: Gathering HAProxy config files 2026-07-08T16:57:57.656841234Z INFO: Waiting for node performance related collection to complete ... 2026-07-08T16:57:58.187596148Z No resources found in openshift-etcd namespace. 2026-07-08T16:57:58.476012163Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-08T16:57:58.476098353Z INFO: on-disk MachineConfig config collection complete. 2026-07-08T16:57:58.491197826Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-08T16:57:58.622349868Z Wrote inspect data to must-gather. 2026-07-08T16:57:58.657397328Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-08T16:57:58.697705655Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-08T16:57:58.862968501Z No resources found 2026-07-08T16:57:58.950391113Z ERROR: No running kube-apiserver pods found 2026-07-08T16:57:58.967919697Z INFO: "metallb-operator" not detected. Skipping. 2026-07-08T16:57:58.982113729Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-dvl4l 2026-07-08T16:57:58.982113729Z insights-runtime-extractor-gk7dn 2026-07-08T16:57:58.982113729Z insights-runtime-extractor-w479z 2026-07-08T16:57:58.982113729Z insights-runtime-extractor-xgtdh 2026-07-08T16:57:59.110948863Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-08T16:57:59.116797978Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-08T16:57:59.315225699Z error: the server doesn't have a resource type "performanceprofile" 2026-07-08T16:57:59.526435570Z INFO: Worker host service log collection to complete. 2026-07-08T16:57:59.557043131Z Wrote inspect data to must-gather. 2026-07-08T16:57:59.669014855Z Wrote inspect data to must-gather. 2026-07-08T16:57:59.791750024Z tar: Removing leading `/' from member names 2026-07-08T16:57:59.796485459Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-08T16:57:59.796485459Z See 'oc rsync -h' for help and examples 2026-07-08T16:57:59.799665791Z INFO: HAProxy config collection complete. 2026-07-08T16:57:59.810597637Z Wrote inspect data to must-gather. 2026-07-08T16:57:59.822617029Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-08T16:57:59.824890777Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-08T16:57:59.839971363Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-08T16:58:00.083595170Z No resources found 2026-07-08T16:58:00.137400054Z Wrote inspect data to must-gather. 2026-07-08T16:58:00.155441603Z INFO: 'previous-logs' folder not found on ip-10-0-138-223.ec2.internal, skipping... 2026-07-08T16:58:00.224214678Z Wrote inspect data to must-gather. 2026-07-08T16:58:00.236112496Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-08T16:58:00.251159747Z Wrote inspect data to must-gather. 2026-07-08T16:58:00.322790070Z Wrote inspect data to must-gather. 2026-07-08T16:58:00.477838037Z error: the server doesn't have a resource type "machineconfigs" 2026-07-08T16:58:00.489835012Z Gathering data for ns/openshift-monitoring... 2026-07-08T16:58:00.532432533Z INFO: 'previous-logs' folder not found on ip-10-0-138-55.ec2.internal, skipping... 2026-07-08T16:58:00.623209820Z Wrote inspect data to must-gather. 2026-07-08T16:58:00.668384544Z INFO: Getting rules from prometheus-k8s-0 2026-07-08T16:58:00.686916686Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-08T16:58:00.862619823Z INFO: 'previous-logs' folder not found on ip-10-0-141-25.ec2.internal, skipping... 2026-07-08T16:58:00.862724443Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-08T16:58:00.862773342Z INFO: Machine Config Daemon termination log collection complete. 2026-07-08T16:58:00.950199164Z Wrote inspect data to must-gather. 2026-07-08T16:58:00.962988583Z Wrote inspect data to must-gather. 2026-07-08T16:58:00.997486029Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-08T16:58:01.135128804Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-08T16:58:01.138182289Z INFO: Getting status/config from prometheus-k8s-0 2026-07-08T16:58:01.265143246Z INFO: INTERCONNECT MODE 2026-07-08T16:58:01.265181026Z INFO: Gathering ovn-kubernetes DBs 2026-07-08T16:58:01.373129702Z Wrote inspect data to must-gather. 2026-07-08T16:58:01.445458693Z INFO: Gathering OVN_Northbound from ovnkube-node-d2c45... 2026-07-08T16:58:01.445736061Z INFO: Gathering OVN_Northbound from ovnkube-node-f9rrd... 2026-07-08T16:58:01.445974894Z INFO: Gathering OVN_Northbound from ovnkube-node-lzzb2... 2026-07-08T16:58:01.446254614Z INFO: Gathering OVN_Southbound from ovnkube-node-d2c45... 2026-07-08T16:58:01.446590388Z INFO: Gathering OVN_Southbound from ovnkube-node-f9rrd... 2026-07-08T16:58:01.446844710Z INFO: Gathering OVN_Southbound from ovnkube-node-lzzb2... 2026-07-08T16:58:01.726372062Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-08T16:58:01.798064393Z tar: Removing leading `/' from member names 2026-07-08T16:58:01.803406849Z tar: Removing leading `/' from member names 2026-07-08T16:58:01.825723998Z Wrote inspect data to must-gather. 2026-07-08T16:58:01.825723998Z tar: Removing leading `/' from member names 2026-07-08T16:58:01.825798790Z tar: Removing leading `/' from member names 2026-07-08T16:58:01.848237766Z tar: Removing leading `/' from member names 2026-07-08T16:58:01.853766439Z tar: Removing leading `/' from member names 2026-07-08T16:58:02.017207002Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-08T16:58:02.096336043Z Wrote inspect data to must-gather. 2026-07-08T16:58:02.195232059Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-08T16:58:02.218375886Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-08T16:58:02.306687595Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-08T16:58:02.358186641Z Wrote inspect data to must-gather. 2026-07-08T16:58:02.360582942Z INFO: Waiting for network log collection to complete ... 2026-07-08T16:58:02.360691406Z INFO: Waiting for ovnk database copies to complete ... 2026-07-08T16:58:02.361192753Z INFO: Copying ovnk databases complete. 2026-07-08T16:58:02.363408031Z 3.3M must-gather/network_logs/ovnk_database_store 2026-07-08T16:58:02.366065497Z ovnk_database_store/ 2026-07-08T16:58:02.366273106Z ovnk_database_store/ovnkube-node-f9rrd_sbdb 2026-07-08T16:58:02.387754642Z ovnk_database_store/ovnkube-node-lzzb2_nbdb 2026-07-08T16:58:02.398920366Z ovnk_database_store/ovnkube-node-d2c45_nbdb 2026-07-08T16:58:02.407914542Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:02.408386189Z ovnk_database_store/ovnkube-node-d2c45_sbdb 2026-07-08T16:58:02.432006853Z ovnk_database_store/ovnkube-node-lzzb2_sbdb 2026-07-08T16:58:02.454715398Z ovnk_database_store/ovnkube-node-f9rrd_nbdb 2026-07-08T16:58:02.468230473Z INFO: Network log collection complete. 2026-07-08T16:58:02.586937082Z Wrote inspect data to must-gather. 2026-07-08T16:58:02.587393545Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-08T16:58:02.645636494Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:02.650748230Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-08T16:58:02.811788514Z W0708 16:58:02.811737 1483 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-08T16:58:02.822985287Z Wrote inspect data to must-gather. 2026-07-08T16:58:02.835643017Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-08T16:58:02.864673558Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:03.016783592Z Gathering data for ns/openshift-network-console... 2026-07-08T16:58:03.020588330Z INFO: Getting status from alertmanager-main-0 2026-07-08T16:58:03.134392190Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:03.137372284Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-08T16:58:03.149741967Z Wrote inspect data to must-gather. 2026-07-08T16:58:03.354156064Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:03.455520448Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-08T16:58:03.458315079Z INFO: Fallback to identify the container image from release info 2026-07-08T16:58:03.515776386Z Gathering data for ns/openshift-console-operator... 2026-07-08T16:58:03.575130270Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:03.578432515Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-08T16:58:03.802043028Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:03.908970242Z Gathering data for ns/openshift-console... 2026-07-08T16:58:04.019572258Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:04.022674485Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-08T16:58:04.075995602Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a 2026-07-08T16:58:04.208987671Z daemonset.apps/perf-node-gather-daemonset created 2026-07-08T16:58:04.211563484Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:04.364557153Z Waiting for performance profile collector pods to become ready: 1 2026-07-08T16:58:04.432705070Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:04.435840168Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-08T16:58:04.502455644Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-08T16:58:04.641963522Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:04.843504964Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-08T16:58:04.847890687Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:04.850498106Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-08T16:58:05.046121964Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:05.142908962Z Gathering data for ns/openshift-dns-operator... 2026-07-08T16:58:05.255190121Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:05.257545677Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-08T16:58:05.441688109Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:05.499470117Z Gathering data for ns/openshift-dns... 2026-07-08T16:58:05.517279641Z Waiting for performance profile collector pods to become ready: 2 2026-07-08T16:58:05.629573597Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:05.632331838Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-08T16:58:05.787840021Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:05.989752324Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:05.994020840Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-08T16:58:06.173723782Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:06.232691785Z Gathering data for ns/openshift-image-registry... 2026-07-08T16:58:06.355598513Z Wrote inspect data to /must-gather/istio. 2026-07-08T16:58:06.651895999Z Waiting for performance profile collector pods to become ready: 3 2026-07-08T16:58:06.760228446Z Gathering data for ns/openshift-ingress-operator... 2026-07-08T16:58:07.083619044Z Done executing Istio gather script 2026-07-08T16:58:07.148609998Z Gathering data for ns/openshift-ingress... 2026-07-08T16:58:07.471792422Z Gathering data for ns/openshift-ingress-canary... 2026-07-08T16:58:07.774669803Z Waiting for performance profile collector pods to become ready: 4 2026-07-08T16:58:07.870496223Z Gathering data for ns/openshift-insights... 2026-07-08T16:58:08.893538075Z Waiting for performance profile collector pods to become ready: 5 2026-07-08T16:58:09.848130911Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-08T16:58:10.025073182Z Waiting for performance profile collector pods to become ready: 6 2026-07-08T16:58:10.146860358Z Gathering data for ns/kserve... 2026-07-08T16:58:11.137899065Z Waiting for performance profile collector pods to become ready: 7 2026-07-08T16:58:11.669366315Z Gathering data for ns/openshift-config... 2026-07-08T16:58:11.977477020Z Gathering data for ns/openshift-config-managed... 2026-07-08T16:58:12.253644186Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-08T16:58:12.398995441Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-08T16:58:12.529054904Z Collecting performance related data for node ip-10-0-138-55.ec2.internal 2026-07-08T16:58:12.536474866Z Collecting performance related data for node ip-10-0-138-223.ec2.internal 2026-07-08T16:58:12.545577186Z Collecting performance related data for node ip-10-0-141-25.ec2.internal 2026-07-08T16:58:12.749715127Z Gathering data for ns/openshift-kube-apiserver... 2026-07-08T16:58:13.062825602Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-08T16:58:13.434125353Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-08T16:58:13.808417098Z Gathering data for ns/openshift-kube-scheduler... 2026-07-08T16:58:14.143144525Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-08T16:58:14.465097798Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-08T16:58:14.802869183Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-08T16:58:14.855227974Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-08T16:58:15.193920925Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-08T16:58:15.813964163Z Gathering data for ns/openshift-multus... 2026-07-08T16:58:17.208824918Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-08T16:58:18.537477414Z Gathering data for ns/openshift-host-network... 2026-07-08T16:58:18.838282607Z Gathering data for ns/openshift-network-diagnostics... 2026-07-08T16:58:19.382444135Z Gathering data for ns/openshift-network-node-identity... 2026-07-08T16:58:19.826622842Z Gathering data for ns/openshift-network-operator... 2026-07-08T16:58:19.859904473Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-08T16:58:20.202120956Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-08T16:58:20.459603739Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-08T16:58:20.612139736Z Collecting kubelet logs for node ip-10-0-138-55.ec2.internal 2026-07-08T16:58:20.614107931Z Collecting kubelet logs for node ip-10-0-138-223.ec2.internal 2026-07-08T16:58:20.615862933Z Collecting kubelet logs for node ip-10-0-141-25.ec2.internal 2026-07-08T16:58:20.964420438Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-hpwtp namespace 2026-07-08T16:58:20.969636760Z INFO: Node performance data collection complete. 2026-07-08T16:58:20.994803178Z Gathering data for ns/openshift-apiserver-operator... 2026-07-08T16:58:21.297872330Z Gathering data for ns/openshift-apiserver... 2026-07-08T16:58:21.670181041Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-08T16:58:21.963447077Z Gathering data for ns/openshift-controller-manager... 2026-07-08T16:58:22.228288366Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-08T16:58:22.853136728Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-08T16:58:23.172525587Z Gathering data for ns/openshift-service-ca-operator... 2026-07-08T16:58:23.491265877Z Gathering data for ns/openshift-service-ca... 2026-07-08T16:58:23.864319503Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-08T16:58:24.781736180Z Wrote inspect data to must-gather. 2026-07-08T16:58:24.781783034Z error: inspection completed with the errors occurred while gathering data: 2026-07-08T16:58:24.781783034Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-08T16:58:24.869287012Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-07-08T16:58:24.914819835Z error: the server doesn't have a resource type "clusters" 2026-07-08T16:58:25.039939488Z Caches written to disk