2026-09-09T21:41:17.543131402Z [disk usage checker] Started 2026-09-09T21:41:17.546575028Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-09-09T21:41:17.743979032Z Gathering data for ns/openshift-cluster-version... 2026-09-09T21:41:17.895806633Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-09T21:41:18.009187756Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-09T21:41:18.097783154Z Gathering data for ns/default... 2026-09-09T21:41:18.469531427Z Gathering data for ns/openshift... 2026-09-09T21:41:18.930492557Z Gathering data for ns/kube-system... 2026-09-09T21:41:19.829738591Z Gathering data for ns/openshift-etcd... 2026-09-09T21:41:20.208000336Z Wrote inspect data to must-gather. 2026-09-09T21:41:20.208033924Z error: inspection completed with the errors occurred while gathering data: 2026-09-09T21:41:20.208033924Z namespaces "assisted-installer" not found 2026-09-09T21:41:20.316313451Z Waiting on subprocesses to finish execution. 2026-09-09T21:41:20.324019491Z INFO: Gathering machine config daemon's old logs from all nodes 2026-09-09T21:41:20.324613609Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-09-09T21:41:20.324982786Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-09-09T21:41:20.333576815Z Executing Istio gather script 2026-09-09T21:41:20.334384007Z INFO: Collecting host service logs for crio 2026-09-09T21:41:20.334721550Z INFO: Collecting host service logs for kubelet 2026-09-09T21:41:20.335071163Z INFO: Collecting host service logs for rpm-ostreed 2026-09-09T21:41:20.335383855Z INFO: Collecting host service logs for ostree-finalize-staged 2026-09-09T21:41:20.335730666Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-09-09T21:41:20.336081133Z INFO: Collecting host service logs for machine-config-daemon-host 2026-09-09T21:41:20.336408990Z INFO: Collecting host service logs for NetworkManager 2026-09-09T21:41:20.336940440Z INFO: Collecting host service logs for openvswitch 2026-09-09T21:41:20.337279507Z INFO: Collecting host service logs for ovs-configuration 2026-09-09T21:41:20.337636334Z INFO: Collecting host service logs for ovsdb-server 2026-09-09T21:41:20.338006653Z INFO: Collecting host service logs for ovs-vswitchd 2026-09-09T21:41:20.338557485Z INFO: Waiting for worker host service log collection to complete ... 2026-09-09T21:41:20.378342788Z INFO: Gathering HAProxy config files 2026-09-09T21:41:20.502906386Z INFO: Waiting for node performance related collection to complete ... 2026-09-09T21:41:21.099096713Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-09-09T21:41:21.099207542Z INFO: on-disk MachineConfig config collection complete. 2026-09-09T21:41:21.170916529Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-npfsq 2026-09-09T21:41:21.170961826Z insights-runtime-extractor-2vtbd 2026-09-09T21:41:21.170980853Z insights-runtime-extractor-gk2zv 2026-09-09T21:41:21.170999093Z insights-runtime-extractor-kzn2d 2026-09-09T21:41:21.215729349Z No resources found 2026-09-09T21:41:21.245640367Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-09-09T21:41:21.270468592Z No resources found in openshift-etcd namespace. 2026-09-09T21:41:21.321467905Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-09-09T21:41:21.392891954Z INFO: "metallb-operator" not detected. Skipping. 2026-09-09T21:41:21.449202422Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-09-09T21:41:21.449202422Z See 'oc rsync -h' for help and examples 2026-09-09T21:41:21.624127925Z error: the server doesn't have a resource type "performanceprofile" 2026-09-09T21:41:21.625947722Z INFO: Found 1 replicas - prometheus-k8s-0 2026-09-09T21:41:21.839945440Z INFO: "sriov-network-operator" not detected. Skipping. 2026-09-09T21:41:21.860268178Z INFO: Worker host service log collection to complete. 2026-09-09T21:41:21.864277559Z Wrote inspect data to must-gather. 2026-09-09T21:41:21.926005321Z ERROR: No running kube-apiserver pods found 2026-09-09T21:41:22.045883305Z INFO: Waiting for HAProxy config collection to complete ... 2026-09-09T21:41:22.298557766Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-09-09T21:41:22.339193273Z Wrote inspect data to must-gather. 2026-09-09T21:41:22.533594231Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-09-09T21:41:22.567487484Z Wrote inspect data to must-gather. 2026-09-09T21:41:22.707384558Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-09T21:41:22.724517380Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-09-09T21:41:22.797164520Z Gathering data for ns/openshift-monitoring... 2026-09-09T21:41:22.809618847Z INFO: 'previous-logs' folder not found on ip-10-0-129-24.ec2.internal, skipping... 2026-09-09T21:41:22.839691338Z Wrote inspect data to must-gather. 2026-09-09T21:41:22.899895337Z Wrote inspect data to must-gather. 2026-09-09T21:41:22.975296338Z No resources found 2026-09-09T21:41:22.995708210Z tar: Removing leading `/' from member names 2026-09-09T21:41:23.002201769Z INFO: HAProxy config collection complete. 2026-09-09T21:41:23.028698377Z Wrote inspect data to must-gather. 2026-09-09T21:41:23.109874550Z Wrote inspect data to must-gather. 2026-09-09T21:41:23.124228479Z error: the server doesn't have a resource type "machineconfigs" 2026-09-09T21:41:23.255568181Z INFO: 'previous-logs' folder not found on ip-10-0-140-67.ec2.internal, skipping... 2026-09-09T21:41:23.294346778Z Wrote inspect data to must-gather. 2026-09-09T21:41:23.337453277Z error: the server doesn't have a resource type "machineconfigpools" 2026-09-09T21:41:23.388114879Z INFO: Getting rules from prometheus-k8s-0 2026-09-09T21:41:23.581150570Z INFO: 'previous-logs' folder not found on ip-10-0-142-161.ec2.internal, skipping... 2026-09-09T21:41:23.581261896Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-09-09T21:41:23.581316035Z INFO: Machine Config Daemon termination log collection complete. 2026-09-09T21:41:23.608450974Z Wrote inspect data to must-gather. 2026-09-09T21:41:23.617156569Z Wrote inspect data to must-gather. 2026-09-09T21:41:23.774353696Z error: the server doesn't have a resource type "kubeletconfigs" 2026-09-09T21:41:23.888607088Z Wrote inspect data to must-gather. 2026-09-09T21:41:23.943062175Z INFO: Getting status/config from prometheus-k8s-0 2026-09-09T21:41:24.037504365Z Wrote inspect data to must-gather. 2026-09-09T21:41:24.181066286Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-09-09T21:41:24.219575469Z INFO: INTERCONNECT MODE 2026-09-09T21:41:24.219651600Z INFO: Gathering ovn-kubernetes DBs 2026-09-09T21:41:24.282943336Z Wrote inspect data to must-gather. 2026-09-09T21:41:24.385358351Z INFO: Getting status/flags from prometheus-k8s-0 2026-09-09T21:41:24.403445294Z INFO: Gathering OVN_Northbound from ovnkube-node-82b4g... 2026-09-09T21:41:24.403639918Z INFO: Gathering OVN_Northbound from ovnkube-node-dv7rn... 2026-09-09T21:41:24.403893057Z INFO: Gathering OVN_Northbound from ovnkube-node-mx4j8... 2026-09-09T21:41:24.404210943Z INFO: Gathering OVN_Southbound from ovnkube-node-82b4g... 2026-09-09T21:41:24.404675214Z INFO: Gathering OVN_Southbound from ovnkube-node-dv7rn... 2026-09-09T21:41:24.404979394Z INFO: Gathering OVN_Southbound from ovnkube-node-mx4j8... 2026-09-09T21:41:24.621603050Z Wrote inspect data to must-gather. 2026-09-09T21:41:24.785908898Z tar: Removing leading `/' from member names 2026-09-09T21:41:24.786142869Z tar: Removing leading `/' from member names 2026-09-09T21:41:24.801311971Z tar: Removing leading `/' from member names 2026-09-09T21:41:24.814389170Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-09-09T21:41:24.817716915Z tar: Removing leading `/' from member names 2026-09-09T21:41:24.820518704Z tar: Removing leading `/' from member names 2026-09-09T21:41:24.843310525Z tar: Removing leading `/' from member names 2026-09-09T21:41:24.949686153Z Wrote inspect data to must-gather. 2026-09-09T21:41:25.106927135Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-09-09T21:41:25.163507794Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-09T21:41:25.196545848Z Wrote inspect data to must-gather. 2026-09-09T21:41:25.249617415Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-09-09T21:41:25.309883912Z INFO: Waiting for network log collection to complete ... 2026-09-09T21:41:25.309927218Z INFO: Waiting for ovnk database copies to complete ... 2026-09-09T21:41:25.310281710Z INFO: Copying ovnk databases complete. 2026-09-09T21:41:25.312011541Z 3.9M must-gather/network_logs/ovnk_database_store 2026-09-09T21:41:25.314726699Z ovnk_database_store/ 2026-09-09T21:41:25.314726699Z ovnk_database_store/ovnkube-node-82b4g_sbdb 2026-09-09T21:41:25.341797504Z ovnk_database_store/ovnkube-node-82b4g_nbdb 2026-09-09T21:41:25.352943927Z ovnk_database_store/ovnkube-node-mx4j8_sbdb 2026-09-09T21:41:25.377063780Z ovnk_database_store/ovnkube-node-dv7rn_sbdb 2026-09-09T21:41:25.390451757Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-09-09T21:41:25.400453552Z ovnk_database_store/ovnkube-node-dv7rn_nbdb 2026-09-09T21:41:25.411695222Z ovnk_database_store/ovnkube-node-mx4j8_nbdb 2026-09-09T21:41:25.423674647Z INFO: Network log collection complete. 2026-09-09T21:41:25.449026621Z Wrote inspect data to must-gather. 2026-09-09T21:41:25.481053380Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:25.695302121Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:25.697706048Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-09-09T21:41:25.706648128Z W0909 21:41:25.706609 1508 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-09-09T21:41:25.711646523Z Wrote inspect data to must-gather. 2026-09-09T21:41:25.819686002Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-09T21:41:25.928086909Z INFO: Getting status from alertmanager-main-0 2026-09-09T21:41:25.930523258Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:26.025558836Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-09-09T21:41:26.030581056Z INFO: Fallback to identify the container image from release info 2026-09-09T21:41:26.049433690Z Gathering data for ns/openshift-network-console... 2026-09-09T21:41:26.168907828Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:26.172926859Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-09-09T21:41:26.361594928Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:26.559219835Z Gathering data for ns/openshift-console-operator... 2026-09-09T21:41:26.572939360Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:26.576577435Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-09-09T21:41:26.760552508Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399 2026-09-09T21:41:26.823048003Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:26.902254428Z daemonset.apps/perf-node-gather-daemonset created 2026-09-09T21:41:27.034413440Z Gathering data for ns/openshift-console... 2026-09-09T21:41:27.061981657Z Waiting for performance profile collector pods to become ready: 1 2026-09-09T21:41:27.066474642Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:27.068928255Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-09-09T21:41:27.241482529Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:27.461358686Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:27.463503469Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-09-09T21:41:27.659993136Z Gathering data for ns/openshift-cluster-storage-operator... 2026-09-09T21:41:27.664962841Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:27.739711921Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-09-09T21:41:27.864224850Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:27.866716735Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-09-09T21:41:28.055228718Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:28.219363409Z Waiting for performance profile collector pods to become ready: 2 2026-09-09T21:41:28.252082381Z Gathering data for ns/openshift-dns-operator... 2026-09-09T21:41:28.305664607Z Wrote inspect data to /must-gather/istio. 2026-09-09T21:41:28.598355887Z Gathering data for ns/openshift-dns... 2026-09-09T21:41:29.004293676Z Done executing Istio gather script 2026-09-09T21:41:29.297889957Z Gathering data for ns/openshift-image-registry... 2026-09-09T21:41:29.346472441Z Waiting for performance profile collector pods to become ready: 3 2026-09-09T21:41:29.771384909Z Gathering data for ns/openshift-ingress-operator... 2026-09-09T21:41:30.117086597Z Gathering data for ns/openshift-ingress... 2026-09-09T21:41:30.480918644Z Waiting for performance profile collector pods to become ready: 4 2026-09-09T21:41:30.488573140Z Gathering data for ns/openshift-ingress-canary... 2026-09-09T21:41:30.898676774Z Gathering data for ns/openshift-insights... 2026-09-09T21:41:31.616711868Z Waiting for performance profile collector pods to become ready: 5 2026-09-09T21:41:32.732774553Z Waiting for performance profile collector pods to become ready: 6 2026-09-09T21:41:32.744792267Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-09-09T21:41:33.360892776Z Gathering data for ns/kserve... 2026-09-09T21:41:33.854826657Z Waiting for performance profile collector pods to become ready: 7 2026-09-09T21:41:34.948370763Z Gathering data for ns/openshift-config... 2026-09-09T21:41:34.975546051Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-09-09T21:41:35.236686442Z Collecting performance related data for node ip-10-0-140-67.ec2.internal 2026-09-09T21:41:35.245292015Z Collecting performance related data for node ip-10-0-142-161.ec2.internal 2026-09-09T21:41:35.253205373Z Collecting performance related data for node ip-10-0-129-24.ec2.internal 2026-09-09T21:41:35.292005442Z Gathering data for ns/openshift-config-managed... 2026-09-09T21:41:35.855781699Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-09-09T21:41:36.222470252Z Gathering data for ns/openshift-kube-apiserver... 2026-09-09T21:41:36.562953059Z Gathering data for ns/openshift-kube-controller-manager... 2026-09-09T21:41:36.944296732Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-09-09T21:41:37.279389673Z Gathering data for ns/openshift-kube-scheduler... 2026-09-09T21:41:37.559716970Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-09-09T21:41:37.755513678Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70 2026-09-09T21:41:37.907809283Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-09-09T21:41:38.319027734Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-09-09T21:41:38.668474786Z Gathering data for ns/openshift-user-workload-monitoring... 2026-09-09T21:41:39.164600296Z Gathering data for ns/openshift-multus... 2026-09-09T21:41:40.663563392Z Gathering data for ns/openshift-ovn-kubernetes... 2026-09-09T21:41:41.963914549Z Gathering data for ns/openshift-host-network... 2026-09-09T21:41:42.314461627Z Gathering data for ns/openshift-network-diagnostics... 2026-09-09T21:41:42.760212774Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-09-09T21:41:42.898081466Z Gathering data for ns/openshift-network-node-identity... 2026-09-09T21:41:43.361807141Z Gathering data for ns/openshift-network-operator... 2026-09-09T21:41:43.390188524Z Collecting kubelet logs for node ip-10-0-140-67.ec2.internal 2026-09-09T21:41:43.391959558Z Collecting kubelet logs for node ip-10-0-142-161.ec2.internal 2026-09-09T21:41:43.393700468Z Collecting kubelet logs for node ip-10-0-129-24.ec2.internal 2026-09-09T21:41:43.772204476Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-kmz7m namespace 2026-09-09T21:41:43.778216579Z INFO: Node performance data collection complete. 2026-09-09T21:41:43.853100368Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-09-09T21:41:44.141649106Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-09-09T21:41:44.620468622Z Gathering data for ns/openshift-apiserver-operator... 2026-09-09T21:41:44.953447423Z Gathering data for ns/openshift-apiserver... 2026-09-09T21:41:45.435889963Z Gathering data for ns/openshift-controller-manager-operator... 2026-09-09T21:41:45.720616229Z Gathering data for ns/openshift-controller-manager... 2026-09-09T21:41:46.041585630Z Gathering data for ns/openshift-cluster-samples-operator... 2026-09-09T21:41:46.702824917Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-09-09T21:41:47.040245785Z Gathering data for ns/openshift-service-ca-operator... 2026-09-09T21:41:47.383059458Z Gathering data for ns/openshift-service-ca... 2026-09-09T21:41:47.745003627Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-09-09T21:41:47.765168523Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70 2026-09-09T21:41:48.653843285Z Wrote inspect data to must-gather. 2026-09-09T21:41:48.653883587Z error: inspection completed with the errors occurred while gathering data: 2026-09-09T21:41:48.653883587Z skipping gathering secrets/support due to error: secrets "support" not found 2026-09-09T21:41:48.784111600Z error: the server doesn't have a resource type "clusters" 2026-09-09T21:41:48.993638255Z Caches written to disk