2026-09-09T21:47:27.903492208Z [disk usage checker] Started 2026-09-09T21:47:27.906825395Z [disk usage checker] Volume usage percentage: current = 34 ; allowed = 70 2026-09-09T21:47:28.194256750Z Gathering data for ns/openshift-cluster-version... 2026-09-09T21:47:28.331468022Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-09T21:47:28.426638468Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-09T21:47:28.510924024Z Gathering data for ns/default... 2026-09-09T21:47:28.840220877Z Gathering data for ns/openshift... 2026-09-09T21:47:29.296610667Z Gathering data for ns/kube-system... 2026-09-09T21:47:30.063562651Z Gathering data for ns/openshift-etcd... 2026-09-09T21:47:30.387676767Z Wrote inspect data to must-gather. 2026-09-09T21:47:30.387741502Z error: inspection completed with the errors occurred while gathering data: 2026-09-09T21:47:30.387741502Z namespaces "assisted-installer" not found 2026-09-09T21:47:30.902028746Z Waiting on subprocesses to finish execution. 2026-09-09T21:47:30.910433830Z Executing Istio gather script 2026-09-09T21:47:30.915864115Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-09-09T21:47:30.919672142Z INFO: Collecting host service logs for crio 2026-09-09T21:47:30.919985088Z INFO: Collecting host service logs for kubelet 2026-09-09T21:47:30.920293615Z INFO: Collecting host service logs for rpm-ostreed 2026-09-09T21:47:30.920584887Z INFO: Collecting host service logs for ostree-finalize-staged 2026-09-09T21:47:30.920962332Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-09-09T21:47:30.921253405Z INFO: Collecting host service logs for machine-config-daemon-host 2026-09-09T21:47:30.921581683Z INFO: Collecting host service logs for NetworkManager 2026-09-09T21:47:30.921914874Z INFO: Collecting host service logs for openvswitch 2026-09-09T21:47:30.922214921Z INFO: Collecting host service logs for ovs-configuration 2026-09-09T21:47:30.922498514Z INFO: Collecting host service logs for ovsdb-server 2026-09-09T21:47:30.922822471Z INFO: Collecting host service logs for ovs-vswitchd 2026-09-09T21:47:30.923179511Z INFO: Waiting for worker host service log collection to complete ... 2026-09-09T21:47:30.936818451Z INFO: Gathering HAProxy config files 2026-09-09T21:47:30.939433171Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-09-09T21:47:30.939745210Z INFO: Gathering machine config daemon's old logs from all nodes 2026-09-09T21:47:30.940120014Z INFO: Waiting for node performance related collection to complete ... 2026-09-09T21:47:31.890387808Z Wrote inspect data to must-gather. 2026-09-09T21:47:32.095292838Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-xnphq 2026-09-09T21:47:32.095340667Z insights-runtime-extractor-bthvc 2026-09-09T21:47:32.095359316Z insights-runtime-extractor-s6gs9 2026-09-09T21:47:32.095379290Z insights-runtime-extractor-wlxqk 2026-09-09T21:47:32.113157029Z No resources found in openshift-etcd namespace. 2026-09-09T21:47:32.160945356Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-09-09T21:47:32.161117409Z INFO: on-disk MachineConfig config collection complete. 2026-09-09T21:47:32.229501504Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-09-09T21:47:32.409039001Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-09-09T21:47:32.449178553Z INFO: "sriov-network-operator" not detected. Skipping. 2026-09-09T21:47:32.450827421Z INFO: Found 1 replicas - prometheus-k8s-0 2026-09-09T21:47:32.521319107Z No resources found 2026-09-09T21:47:32.530685509Z INFO: Worker host service log collection to complete. 2026-09-09T21:47:32.539442859Z INFO: "metallb-operator" not detected. Skipping. 2026-09-09T21:47:32.540219908Z error: the server doesn't have a resource type "performanceprofile" 2026-09-09T21:47:32.556421435Z Wrote inspect data to must-gather. 2026-09-09T21:47:32.767450636Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-09-09T21:47:32.767450636Z See 'oc rsync -h' for help and examples 2026-09-09T21:47:32.803578878Z INFO: Waiting for HAProxy config collection to complete ... 2026-09-09T21:47:32.950146297Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-09-09T21:47:33.011155314Z [disk usage checker] Volume usage percentage: current = 34 ; allowed = 70 2026-09-09T21:47:33.094342073Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-09T21:47:33.141243669Z Wrote inspect data to must-gather. 2026-09-09T21:47:33.148959734Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-09-09T21:47:33.181977548Z Wrote inspect data to must-gather. 2026-09-09T21:47:33.365209717Z No resources found 2026-09-09T21:47:33.396528571Z Wrote inspect data to must-gather. 2026-09-09T21:47:33.505229852Z ERROR: No running kube-apiserver pods found 2026-09-09T21:47:33.543546461Z Wrote inspect data to must-gather. 2026-09-09T21:47:33.713108610Z Wrote inspect data to must-gather. 2026-09-09T21:47:33.715018919Z tar: Removing leading `/' from member names 2026-09-09T21:47:33.722493319Z INFO: HAProxy config collection complete. 2026-09-09T21:47:33.741222312Z INFO: 'previous-logs' folder not found on ip-10-0-130-40.ec2.internal, skipping... 2026-09-09T21:47:33.748748477Z Gathering data for ns/openshift-monitoring... 2026-09-09T21:47:33.841797120Z Wrote inspect data to must-gather. 2026-09-09T21:47:33.955614811Z error: the server doesn't have a resource type "machineconfigs" 2026-09-09T21:47:34.016286149Z INFO: Getting rules from prometheus-k8s-0 2026-09-09T21:47:34.077580725Z INFO: 'previous-logs' folder not found on ip-10-0-136-87.ec2.internal, skipping... 2026-09-09T21:47:34.169796072Z error: the server doesn't have a resource type "machineconfigpools" 2026-09-09T21:47:34.177415079Z Wrote inspect data to must-gather. 2026-09-09T21:47:34.375490973Z INFO: 'previous-logs' folder not found on ip-10-0-137-145.ec2.internal, skipping... 2026-09-09T21:47:34.375596931Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-09-09T21:47:34.375642346Z INFO: Machine Config Daemon termination log collection complete. 2026-09-09T21:47:34.463802026Z Wrote inspect data to must-gather. 2026-09-09T21:47:34.474438983Z Wrote inspect data to must-gather. 2026-09-09T21:47:34.510498814Z INFO: Getting status/config from prometheus-k8s-0 2026-09-09T21:47:34.652062480Z error: the server doesn't have a resource type "kubeletconfigs" 2026-09-09T21:47:34.664645562Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-09-09T21:47:34.795095600Z INFO: INTERCONNECT MODE 2026-09-09T21:47:34.795129204Z INFO: Gathering ovn-kubernetes DBs 2026-09-09T21:47:34.876000720Z Wrote inspect data to must-gather. 2026-09-09T21:47:34.965030846Z INFO: Gathering OVN_Northbound from ovnkube-node-fc6xb... 2026-09-09T21:47:34.965246748Z INFO: Gathering OVN_Northbound from ovnkube-node-jj6bk... 2026-09-09T21:47:34.965479678Z INFO: Gathering OVN_Northbound from ovnkube-node-zp8rd... 2026-09-09T21:47:34.965785197Z INFO: Gathering OVN_Southbound from ovnkube-node-fc6xb... 2026-09-09T21:47:34.966075606Z INFO: Gathering OVN_Southbound from ovnkube-node-jj6bk... 2026-09-09T21:47:34.966377211Z INFO: Gathering OVN_Southbound from ovnkube-node-zp8rd... 2026-09-09T21:47:35.071790677Z INFO: Getting status/flags from prometheus-k8s-0 2026-09-09T21:47:35.249984019Z Wrote inspect data to must-gather. 2026-09-09T21:47:35.266215712Z tar: Removing leading `/' from member names 2026-09-09T21:47:35.266284382Z tar: Removing leading `/' from member names 2026-09-09T21:47:35.269920571Z tar: Removing leading `/' from member names 2026-09-09T21:47:35.324875154Z tar: Removing leading `/' from member names 2026-09-09T21:47:35.353884160Z tar: Removing leading `/' from member names 2026-09-09T21:47:35.371470490Z tar: Removing leading `/' from member names 2026-09-09T21:47:35.442092215Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-09-09T21:47:35.487342716Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-09T21:47:35.548285769Z Wrote inspect data to must-gather. 2026-09-09T21:47:35.778480931Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-09-09T21:47:35.819321825Z Wrote inspect data to must-gather. 2026-09-09T21:47:35.896512612Z INFO: Waiting for network log collection to complete ... 2026-09-09T21:47:35.896664915Z INFO: Waiting for ovnk database copies to complete ... 2026-09-09T21:47:35.896932814Z INFO: Copying ovnk databases complete. 2026-09-09T21:47:35.898612297Z 4.8M must-gather/network_logs/ovnk_database_store 2026-09-09T21:47:35.901634457Z ovnk_database_store/ 2026-09-09T21:47:35.901684572Z ovnk_database_store/ovnkube-node-fc6xb_nbdb 2026-09-09T21:47:35.916454340Z ovnk_database_store/ovnkube-node-fc6xb_sbdb 2026-09-09T21:47:35.955358364Z ovnk_database_store/ovnkube-node-jj6bk_nbdb 2026-09-09T21:47:35.965512344Z ovnk_database_store/ovnkube-node-jj6bk_sbdb 2026-09-09T21:47:35.988028457Z ovnk_database_store/ovnkube-node-zp8rd_sbdb 2026-09-09T21:47:36.018238043Z ovnk_database_store/ovnkube-node-zp8rd_nbdb 2026-09-09T21:47:36.032606686Z INFO: Network log collection complete. 2026-09-09T21:47:36.050853110Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-09-09T21:47:36.054137071Z W0909 21:47:36.054094 1443 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-09-09T21:47:36.054185560Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-09-09T21:47:36.059014028Z Wrote inspect data to must-gather. 2026-09-09T21:47:36.143886930Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-09T21:47:36.271278172Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:36.356828948Z Gathering data for ns/openshift-network-console... 2026-09-09T21:47:36.378822614Z Wrote inspect data to must-gather. 2026-09-09T21:47:36.498059500Z INFO: Getting status from alertmanager-main-0 2026-09-09T21:47:36.529562365Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:36.535138984Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-09-09T21:47:36.717945361Z Wrote inspect data to must-gather. 2026-09-09T21:47:36.761467775Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:36.874564582Z Gathering data for ns/openshift-console-operator... 2026-09-09T21:47:37.008663833Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:37.012494179Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-09-09T21:47:37.031607423Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-09-09T21:47:37.034825018Z INFO: Fallback to identify the container image from release info 2026-09-09T21:47:37.227062690Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:37.313029481Z Gathering data for ns/openshift-console... 2026-09-09T21:47:37.432655517Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:37.438012856Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-09-09T21:47:37.676146121Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:37.684121581Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399 2026-09-09T21:47:37.832364723Z daemonset.apps/perf-node-gather-daemonset created 2026-09-09T21:47:37.914012914Z Gathering data for ns/openshift-cluster-storage-operator... 2026-09-09T21:47:37.914642340Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:37.917925859Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-09-09T21:47:37.990931714Z Waiting for performance profile collector pods to become ready: 1 2026-09-09T21:47:38.074560464Z [disk usage checker] Volume usage percentage: current = 34 ; allowed = 70 2026-09-09T21:47:38.108724792Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:38.348374370Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:38.352552713Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-09-09T21:47:38.505970312Z Gathering data for ns/openshift-dns-operator... 2026-09-09T21:47:38.550759611Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:38.754627707Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:38.758206292Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-09-09T21:47:38.866628288Z Gathering data for ns/openshift-dns... 2026-09-09T21:47:38.945749810Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:39.129231283Z Waiting for performance profile collector pods to become ready: 2 2026-09-09T21:47:39.145787701Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:47:39.606637364Z Gathering data for ns/openshift-image-registry... 2026-09-09T21:47:39.872080833Z Done executing Istio gather script 2026-09-09T21:47:40.123182773Z Gathering data for ns/openshift-ingress-operator... 2026-09-09T21:47:40.266025156Z Waiting for performance profile collector pods to become ready: 3 2026-09-09T21:47:40.471799740Z Gathering data for ns/openshift-ingress... 2026-09-09T21:47:40.807589516Z Gathering data for ns/openshift-ingress-canary... 2026-09-09T21:47:41.200690107Z Gathering data for ns/openshift-insights... 2026-09-09T21:47:41.381220264Z Waiting for performance profile collector pods to become ready: 4 2026-09-09T21:47:42.499828268Z Waiting for performance profile collector pods to become ready: 5 2026-09-09T21:47:43.079179247Z [disk usage checker] Volume usage percentage: current = 34 ; allowed = 70 2026-09-09T21:47:43.623366088Z Waiting for performance profile collector pods to become ready: 6 2026-09-09T21:47:43.687065403Z Gathering data for ns/kserve... 2026-09-09T21:47:44.762743126Z Waiting for performance profile collector pods to become ready: 7 2026-09-09T21:47:45.886063761Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-09-09T21:47:46.099295303Z Gathering data for ns/openshift-config... 2026-09-09T21:47:46.172465645Z Collecting performance related data for node ip-10-0-130-40.ec2.internal 2026-09-09T21:47:46.180501169Z Collecting performance related data for node ip-10-0-137-145.ec2.internal 2026-09-09T21:47:46.187877903Z Collecting performance related data for node ip-10-0-136-87.ec2.internal 2026-09-09T21:47:46.409959408Z Gathering data for ns/openshift-config-managed... 2026-09-09T21:47:46.935234678Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-09-09T21:47:47.454329925Z Gathering data for ns/openshift-kube-apiserver... 2026-09-09T21:47:47.823466626Z Gathering data for ns/openshift-kube-controller-manager... 2026-09-09T21:47:48.085302615Z [disk usage checker] Volume usage percentage: current = 34 ; allowed = 70 2026-09-09T21:47:48.212155396Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-09-09T21:47:48.531056040Z Gathering data for ns/openshift-kube-scheduler... 2026-09-09T21:47:48.844473898Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-09-09T21:47:49.248051130Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-09-09T21:47:49.669439302Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-09-09T21:47:50.043569419Z Gathering data for ns/openshift-user-workload-monitoring... 2026-09-09T21:47:50.577824367Z Gathering data for ns/openshift-multus... 2026-09-09T21:47:52.356193135Z Gathering data for ns/openshift-ovn-kubernetes... 2026-09-09T21:47:53.089923854Z [disk usage checker] Volume usage percentage: current = 34 ; allowed = 70 2026-09-09T21:47:53.915433013Z Gathering data for ns/openshift-host-network... 2026-09-09T21:47:54.262656887Z Gathering data for ns/openshift-network-diagnostics... 2026-09-09T21:47:54.830725788Z Collecting kubelet logs for node ip-10-0-130-40.ec2.internal 2026-09-09T21:47:54.832596139Z Collecting kubelet logs for node ip-10-0-137-145.ec2.internal 2026-09-09T21:47:54.835316128Z Collecting kubelet logs for node ip-10-0-136-87.ec2.internal 2026-09-09T21:47:54.871993429Z Gathering data for ns/openshift-network-node-identity... 2026-09-09T21:47:55.295538337Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-qw289 namespace 2026-09-09T21:47:55.302917580Z INFO: Node performance data collection complete. 2026-09-09T21:47:55.393422089Z Gathering data for ns/openshift-network-operator... 2026-09-09T21:47:55.841360276Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-09-09T21:47:56.122740766Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-09-09T21:47:56.691012799Z Gathering data for ns/openshift-apiserver-operator... 2026-09-09T21:47:57.010782166Z Gathering data for ns/openshift-apiserver... 2026-09-09T21:47:57.545644541Z Gathering data for ns/openshift-controller-manager-operator... 2026-09-09T21:47:57.848527649Z Gathering data for ns/openshift-controller-manager... 2026-09-09T21:47:58.095080215Z [disk usage checker] Volume usage percentage: current = 34 ; allowed = 70 2026-09-09T21:47:58.185961475Z Gathering data for ns/openshift-cluster-samples-operator... 2026-09-09T21:47:58.855466935Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-09-09T21:47:59.218159626Z Gathering data for ns/openshift-service-ca-operator... 2026-09-09T21:47:59.554395569Z Gathering data for ns/openshift-service-ca... 2026-09-09T21:47:59.951960655Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-09-09T21:48:00.892389085Z Wrote inspect data to must-gather. 2026-09-09T21:48:00.892437419Z error: inspection completed with the errors occurred while gathering data: 2026-09-09T21:48:00.892437419Z skipping gathering secrets/support due to error: secrets "support" not found 2026-09-09T21:48:01.036211473Z error: the server doesn't have a resource type "clusters" 2026-09-09T21:48:01.187293293Z Caches written to disk