2026-07-29T17:07:34.141098772Z [disk usage checker] Started 2026-07-29T17:07:34.144700213Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:07:34.333425353Z Gathering data for ns/openshift-cluster-version... 2026-07-29T17:07:34.480411228Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-29T17:07:34.574997742Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-29T17:07:34.664717134Z Gathering data for ns/default... 2026-07-29T17:07:34.964937399Z Gathering data for ns/openshift... 2026-07-29T17:07:35.365704858Z Gathering data for ns/kube-system... 2026-07-29T17:07:36.069660671Z Gathering data for ns/openshift-etcd... 2026-07-29T17:07:36.380467882Z Wrote inspect data to must-gather. 2026-07-29T17:07:36.380564540Z error: inspection completed with the errors occurred while gathering data: 2026-07-29T17:07:36.380564540Z namespaces "assisted-installer" not found 2026-07-29T17:07:36.915236965Z Waiting on subprocesses to finish execution. 2026-07-29T17:07:36.922199116Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-29T17:07:36.922682003Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-29T17:07:36.925050531Z INFO: Gathering HAProxy config files 2026-07-29T17:07:36.927067542Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-29T17:07:36.927624273Z Executing Istio gather script 2026-07-29T17:07:36.931921183Z INFO: Collecting host service logs for crio 2026-07-29T17:07:36.932270307Z INFO: Collecting host service logs for kubelet 2026-07-29T17:07:36.932616442Z INFO: Collecting host service logs for rpm-ostreed 2026-07-29T17:07:36.932969692Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-29T17:07:36.933313006Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-29T17:07:36.933651809Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-29T17:07:36.934080786Z INFO: Collecting host service logs for NetworkManager 2026-07-29T17:07:36.934414656Z INFO: Collecting host service logs for openvswitch 2026-07-29T17:07:36.934754955Z INFO: Collecting host service logs for ovs-configuration 2026-07-29T17:07:36.935119666Z INFO: Collecting host service logs for ovsdb-server 2026-07-29T17:07:36.935477925Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-29T17:07:36.935904923Z INFO: Waiting for worker host service log collection to complete ... 2026-07-29T17:07:37.125944861Z INFO: Waiting for node performance related collection to complete ... 2026-07-29T17:07:37.359486057Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-29T17:07:37.911844044Z No resources found in openshift-etcd namespace. 2026-07-29T17:07:38.024933898Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-29T17:07:38.025102929Z INFO: on-disk MachineConfig config collection complete. 2026-07-29T17:07:38.037454178Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-xc5fq 2026-07-29T17:07:38.037499859Z insights-runtime-extractor-6ffg5 2026-07-29T17:07:38.037519312Z insights-runtime-extractor-f65z4 2026-07-29T17:07:38.037537716Z insights-runtime-extractor-jwv5z 2026-07-29T17:07:38.075714264Z Wrote inspect data to must-gather. 2026-07-29T17:07:38.131005333Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-29T17:07:38.221437247Z No resources found 2026-07-29T17:07:38.236015583Z ERROR: No running kube-apiserver pods found 2026-07-29T17:07:38.363850936Z error: the server doesn't have a resource type "performanceprofile" 2026-07-29T17:07:38.444565105Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-29T17:07:38.488522165Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-29T17:07:38.502962061Z INFO: Worker host service log collection to complete. 2026-07-29T17:07:38.591949923Z INFO: "metallb-operator" not detected. Skipping. 2026-07-29T17:07:38.636932894Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-29T17:07:38.743112518Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-29T17:07:38.783173189Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-29T17:07:38.783173189Z See 'oc rsync -h' for help and examples 2026-07-29T17:07:39.029207425Z Wrote inspect data to must-gather. 2026-07-29T17:07:39.166886083Z Wrote inspect data to must-gather. 2026-07-29T17:07:39.290849984Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-29T17:07:39.342949138Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:07:39.359282943Z No resources found 2026-07-29T17:07:39.408194277Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-29T17:07:39.486815708Z Wrote inspect data to must-gather. 2026-07-29T17:07:39.554721537Z INFO: 'previous-logs' folder not found on ip-10-0-131-25.ec2.internal, skipping... 2026-07-29T17:07:39.627966835Z Wrote inspect data to must-gather. 2026-07-29T17:07:39.641022010Z Wrote inspect data to must-gather. 2026-07-29T17:07:39.719663812Z tar: Removing leading `/' from member names 2026-07-29T17:07:39.725324195Z INFO: HAProxy config collection complete. 2026-07-29T17:07:39.751533599Z error: the server doesn't have a resource type "machineconfigs" 2026-07-29T17:07:39.769639269Z Gathering data for ns/openshift-monitoring... 2026-07-29T17:07:39.833511986Z Wrote inspect data to must-gather. 2026-07-29T17:07:39.876584962Z INFO: Getting rules from prometheus-k8s-0 2026-07-29T17:07:39.980038261Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-29T17:07:40.006514321Z INFO: 'previous-logs' folder not found on ip-10-0-141-139.ec2.internal, skipping... 2026-07-29T17:07:40.136627298Z Wrote inspect data to must-gather. 2026-07-29T17:07:40.272225626Z Wrote inspect data to must-gather. 2026-07-29T17:07:40.323231259Z INFO: 'previous-logs' folder not found on ip-10-0-141-161.ec2.internal, skipping... 2026-07-29T17:07:40.323270821Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-29T17:07:40.323296367Z INFO: Machine Config Daemon termination log collection complete. 2026-07-29T17:07:40.377488772Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-29T17:07:40.381290637Z INFO: Getting status/config from prometheus-k8s-0 2026-07-29T17:07:40.446105505Z Wrote inspect data to must-gather. 2026-07-29T17:07:40.455697022Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-29T17:07:40.684509429Z Wrote inspect data to must-gather. 2026-07-29T17:07:40.719728247Z Wrote inspect data to must-gather. 2026-07-29T17:07:40.837216786Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-29T17:07:40.930460950Z Wrote inspect data to must-gather. 2026-07-29T17:07:41.031448912Z INFO: INTERCONNECT MODE 2026-07-29T17:07:41.031483602Z INFO: Gathering ovn-kubernetes DBs 2026-07-29T17:07:41.078273004Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-29T17:07:41.159744310Z Wrote inspect data to must-gather. 2026-07-29T17:07:41.223320753Z INFO: Gathering OVN_Northbound from ovnkube-node-8nfw4... 2026-07-29T17:07:41.223732829Z INFO: Gathering OVN_Northbound from ovnkube-node-mxwxc... 2026-07-29T17:07:41.224036154Z INFO: Gathering OVN_Northbound from ovnkube-node-xczd9... 2026-07-29T17:07:41.224345751Z INFO: Gathering OVN_Southbound from ovnkube-node-8nfw4... 2026-07-29T17:07:41.224630503Z INFO: Gathering OVN_Southbound from ovnkube-node-mxwxc... 2026-07-29T17:07:41.224936355Z INFO: Gathering OVN_Southbound from ovnkube-node-xczd9... 2026-07-29T17:07:41.426295074Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-29T17:07:41.461001835Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-29T17:07:41.562096138Z Wrote inspect data to must-gather. 2026-07-29T17:07:41.581683466Z tar: Removing leading `/' from member names 2026-07-29T17:07:41.585584982Z tar: Removing leading `/' from member names 2026-07-29T17:07:41.585584982Z tar: Removing leading `/' from member names 2026-07-29T17:07:41.588416131Z tar: Removing leading `/' from member names 2026-07-29T17:07:41.608051069Z tar: Removing leading `/' from member names 2026-07-29T17:07:41.615968079Z tar: Removing leading `/' from member names 2026-07-29T17:07:41.668239907Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-29T17:07:41.807176174Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-29T17:07:41.834187595Z Wrote inspect data to must-gather. 2026-07-29T17:07:41.895644873Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:42.120286423Z W0729 17:07:42.120238 1465 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-29T17:07:42.127539663Z Wrote inspect data to must-gather. 2026-07-29T17:07:42.171148509Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:42.173391237Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-29T17:07:42.207916779Z INFO: Waiting for network log collection to complete ... 2026-07-29T17:07:42.208031978Z INFO: Waiting for ovnk database copies to complete ... 2026-07-29T17:07:42.208556899Z INFO: Copying ovnk databases complete. 2026-07-29T17:07:42.210457925Z 2.7M must-gather/network_logs/ovnk_database_store 2026-07-29T17:07:42.213034381Z ovnk_database_store/ 2026-07-29T17:07:42.213034381Z ovnk_database_store/ovnkube-node-mxwxc_nbdb 2026-07-29T17:07:42.222290691Z ovnk_database_store/ovnkube-node-8nfw4_nbdb 2026-07-29T17:07:42.228180472Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-29T17:07:42.228326862Z ovnk_database_store/ovnkube-node-8nfw4_sbdb 2026-07-29T17:07:42.249314361Z ovnk_database_store/ovnkube-node-xczd9_sbdb 2026-07-29T17:07:42.269658808Z ovnk_database_store/ovnkube-node-xczd9_nbdb 2026-07-29T17:07:42.277425404Z ovnk_database_store/ovnkube-node-mxwxc_sbdb 2026-07-29T17:07:42.300057882Z INFO: Network log collection complete. 2026-07-29T17:07:42.300733467Z INFO: Getting status from alertmanager-main-0 2026-07-29T17:07:42.412260194Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:42.436718101Z Gathering data for ns/openshift-network-console... 2026-07-29T17:07:42.483175565Z Wrote inspect data to must-gather. 2026-07-29T17:07:42.624444228Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:42.626635405Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-29T17:07:42.770867310Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-29T17:07:42.774151224Z INFO: Fallback to identify the container image from release info 2026-07-29T17:07:42.843970554Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:42.905480218Z Gathering data for ns/openshift-console-operator... 2026-07-29T17:07:43.052441056Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:43.054940770Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-29T17:07:43.264456788Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:43.298854046Z Gathering data for ns/openshift-console... 2026-07-29T17:07:43.461495331Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:43.464168748Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-29T17:07:43.649173980Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:43.683103230Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 2026-07-29T17:07:43.837315953Z daemonset.apps/perf-node-gather-daemonset created 2026-07-29T17:07:43.846983801Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-29T17:07:43.877904331Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:43.880687686Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-29T17:07:43.992601505Z Waiting for performance profile collector pods to become ready: 1 2026-07-29T17:07:44.067357769Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:44.293723714Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:44.297133643Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-29T17:07:44.349846647Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:07:44.457616606Z Gathering data for ns/openshift-dns-operator... 2026-07-29T17:07:44.511863333Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:44.718360915Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:44.722218559Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-29T17:07:44.774865621Z Gathering data for ns/openshift-dns... 2026-07-29T17:07:44.885886046Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:45.064256639Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:45.067078261Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-29T17:07:45.123600800Z Waiting for performance profile collector pods to become ready: 2 2026-07-29T17:07:45.244191446Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:45.435882374Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:45.439900520Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-29T17:07:45.559594422Z Gathering data for ns/openshift-image-registry... 2026-07-29T17:07:45.618914128Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:45.805006195Z Wrote inspect data to /must-gather/istio. 2026-07-29T17:07:46.070910265Z Gathering data for ns/openshift-ingress-operator... 2026-07-29T17:07:46.262687711Z Waiting for performance profile collector pods to become ready: 3 2026-07-29T17:07:46.454139417Z Gathering data for ns/openshift-ingress... 2026-07-29T17:07:46.544291496Z Done executing Istio gather script 2026-07-29T17:07:46.788496579Z Gathering data for ns/openshift-ingress-canary... 2026-07-29T17:07:47.232083612Z Gathering data for ns/openshift-insights... 2026-07-29T17:07:47.377823601Z Waiting for performance profile collector pods to become ready: 4 2026-07-29T17:07:48.495565900Z Waiting for performance profile collector pods to become ready: 5 2026-07-29T17:07:49.354724408Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:07:49.501958779Z Gathering data for ns/kserve... 2026-07-29T17:07:49.620980453Z Waiting for performance profile collector pods to become ready: 6 2026-07-29T17:07:50.749500714Z Waiting for performance profile collector pods to become ready: 7 2026-07-29T17:07:50.774149784Z Gathering data for ns/openshift-config... 2026-07-29T17:07:51.052261850Z Gathering data for ns/openshift-config-managed... 2026-07-29T17:07:51.473548989Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-29T17:07:51.789496065Z Gathering data for ns/openshift-kube-apiserver... 2026-07-29T17:07:51.892616320Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-29T17:07:52.124046057Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-29T17:07:52.180459322Z Collecting performance related data for node ip-10-0-141-161.ec2.internal 2026-07-29T17:07:52.188497319Z Collecting performance related data for node ip-10-0-131-25.ec2.internal 2026-07-29T17:07:52.198239785Z Collecting performance related data for node ip-10-0-141-139.ec2.internal 2026-07-29T17:07:52.475141169Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-29T17:07:52.886813015Z Gathering data for ns/openshift-kube-scheduler... 2026-07-29T17:07:53.233269307Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-29T17:07:53.616301260Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-29T17:07:53.989744028Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-29T17:07:54.293122309Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-29T17:07:54.359958567Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:07:54.857003945Z Gathering data for ns/openshift-multus... 2026-07-29T17:07:56.314269867Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-29T17:07:57.643306944Z Gathering data for ns/openshift-host-network... 2026-07-29T17:07:57.968957532Z Gathering data for ns/openshift-network-diagnostics... 2026-07-29T17:07:58.514746666Z Gathering data for ns/openshift-network-node-identity... 2026-07-29T17:07:58.951716157Z Gathering data for ns/openshift-network-operator... 2026-07-29T17:07:59.348018949Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-29T17:07:59.365395556Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-29T17:07:59.634336692Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-29T17:08:00.133833057Z Gathering data for ns/openshift-apiserver-operator... 2026-07-29T17:08:00.437551287Z Gathering data for ns/openshift-apiserver... 2026-07-29T17:08:00.537294685Z Collecting kubelet logs for node ip-10-0-141-161.ec2.internal 2026-07-29T17:08:00.539049818Z Collecting kubelet logs for node ip-10-0-131-25.ec2.internal 2026-07-29T17:08:00.540717545Z Collecting kubelet logs for node ip-10-0-141-139.ec2.internal 2026-07-29T17:08:00.917451902Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-jd8ht namespace 2026-07-29T17:08:00.922711977Z INFO: Node performance data collection complete. 2026-07-29T17:08:00.927537740Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-29T17:08:01.234718308Z Gathering data for ns/openshift-controller-manager... 2026-07-29T17:08:01.517378480Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-29T17:08:02.192325359Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-29T17:08:02.481554041Z Gathering data for ns/openshift-service-ca-operator... 2026-07-29T17:08:02.811042558Z Gathering data for ns/openshift-service-ca... 2026-07-29T17:08:03.169208785Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-29T17:08:04.065979722Z Wrote inspect data to must-gather. 2026-07-29T17:08:04.066026130Z error: inspection completed with the errors occurred while gathering data: 2026-07-29T17:08:04.066026130Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-29T17:08:04.195283272Z error: the server doesn't have a resource type "clusters" 2026-07-29T17:08:04.339788134Z Caches written to disk