2026-09-10T12:47:59.703984489Z [disk usage checker] Started 2026-09-10T12:47:59.707367525Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-09-10T12:47:59.904310751Z Gathering data for ns/openshift-cluster-version... 2026-09-10T12:48:00.044556753Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-10T12:48:00.126287917Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-10T12:48:00.207244391Z Gathering data for ns/default... 2026-09-10T12:48:00.483192968Z Gathering data for ns/openshift... 2026-09-10T12:48:00.862282930Z Gathering data for ns/kube-system... 2026-09-10T12:48:01.519042289Z Gathering data for ns/openshift-etcd... 2026-09-10T12:48:01.792752548Z Wrote inspect data to must-gather. 2026-09-10T12:48:01.792785316Z error: inspection completed with the errors occurred while gathering data: 2026-09-10T12:48:01.792785316Z namespaces "assisted-installer" not found 2026-09-10T12:48:02.173578194Z Waiting on subprocesses to finish execution. 2026-09-10T12:48:02.185170534Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-09-10T12:48:02.186623170Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-09-10T12:48:02.189755974Z INFO: Gathering machine config daemon's old logs from all nodes 2026-09-10T12:48:02.189875982Z Executing Istio gather script 2026-09-10T12:48:02.199908202Z INFO: Gathering HAProxy config files 2026-09-10T12:48:02.232484172Z INFO: Collecting host service logs for crio 2026-09-10T12:48:02.232812952Z INFO: Collecting host service logs for kubelet 2026-09-10T12:48:02.233161266Z INFO: Collecting host service logs for rpm-ostreed 2026-09-10T12:48:02.233486165Z INFO: Collecting host service logs for ostree-finalize-staged 2026-09-10T12:48:02.233806372Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-09-10T12:48:02.234163599Z INFO: Collecting host service logs for machine-config-daemon-host 2026-09-10T12:48:02.234485938Z INFO: Collecting host service logs for NetworkManager 2026-09-10T12:48:02.234803273Z INFO: Collecting host service logs for openvswitch 2026-09-10T12:48:02.235136537Z INFO: Collecting host service logs for ovs-configuration 2026-09-10T12:48:02.235456156Z INFO: Collecting host service logs for ovsdb-server 2026-09-10T12:48:02.235825404Z INFO: Collecting host service logs for ovs-vswitchd 2026-09-10T12:48:02.236308933Z INFO: Waiting for worker host service log collection to complete ... 2026-09-10T12:48:02.371266292Z INFO: Waiting for node performance related collection to complete ... 2026-09-10T12:48:02.595474887Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-tg74b 2026-09-10T12:48:02.595474887Z insights-runtime-extractor-mkszr 2026-09-10T12:48:02.595474887Z insights-runtime-extractor-pqcdq 2026-09-10T12:48:02.595474887Z insights-runtime-extractor-t5mdr 2026-09-10T12:48:02.986448512Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-09-10T12:48:03.168196474Z INFO: Found 1 replicas - prometheus-k8s-0 2026-09-10T12:48:03.172610854Z No resources found 2026-09-10T12:48:03.288200435Z INFO: "sriov-network-operator" not detected. Skipping. 2026-09-10T12:48:03.371397549Z No resources found in openshift-etcd namespace. 2026-09-10T12:48:03.410290330Z INFO: "metallb-operator" not detected. Skipping. 2026-09-10T12:48:03.423822970Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-09-10T12:48:03.544941013Z Wrote inspect data to must-gather. 2026-09-10T12:48:03.557070108Z error: the server doesn't have a resource type "performanceprofile" 2026-09-10T12:48:03.631097264Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-09-10T12:48:03.631097264Z INFO: on-disk MachineConfig config collection complete. 2026-09-10T12:48:03.804799683Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-09-10T12:48:03.804799683Z See 'oc rsync -h' for help and examples 2026-09-10T12:48:03.907622506Z INFO: Worker host service log collection to complete. 2026-09-10T12:48:04.105939228Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-10T12:48:04.163941659Z Wrote inspect data to must-gather. 2026-09-10T12:48:04.202944337Z ERROR: No running kube-apiserver pods found 2026-09-10T12:48:04.302260527Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-09-10T12:48:04.314952083Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-09-10T12:48:04.363892840Z Wrote inspect data to must-gather. 2026-09-10T12:48:04.421413398Z No resources found 2026-09-10T12:48:04.530014441Z Wrote inspect data to must-gather. 2026-09-10T12:48:04.541288060Z INFO: Waiting for HAProxy config collection to complete ... 2026-09-10T12:48:04.610288039Z Wrote inspect data to must-gather. 2026-09-10T12:48:04.651661407Z Wrote inspect data to must-gather. 2026-09-10T12:48:04.669561891Z INFO: 'previous-logs' folder not found on ip-10-0-130-221.ec2.internal, skipping... 2026-09-10T12:48:04.716391122Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-09-10T12:48:04.720947530Z Wrote inspect data to must-gather. 2026-09-10T12:48:04.858254446Z tar: Removing leading `/' from member names 2026-09-10T12:48:04.862083069Z Gathering data for ns/openshift-monitoring... 2026-09-10T12:48:04.863063101Z INFO: HAProxy config collection complete. 2026-09-10T12:48:04.932033926Z error: the server doesn't have a resource type "machineconfigs" 2026-09-10T12:48:04.950451753Z Wrote inspect data to must-gather. 2026-09-10T12:48:04.957171166Z INFO: Getting rules from prometheus-k8s-0 2026-09-10T12:48:04.958997100Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-09-10T12:48:05.002372498Z INFO: 'previous-logs' folder not found on ip-10-0-136-197.ec2.internal, skipping... 2026-09-10T12:48:05.109725001Z error: the server doesn't have a resource type "machineconfigpools" 2026-09-10T12:48:05.216284244Z Wrote inspect data to must-gather. 2026-09-10T12:48:05.264467181Z INFO: 'previous-logs' folder not found on ip-10-0-143-110.ec2.internal, skipping... 2026-09-10T12:48:05.264503410Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-09-10T12:48:05.264503410Z INFO: Machine Config Daemon termination log collection complete. 2026-09-10T12:48:05.347407631Z Wrote inspect data to must-gather. 2026-09-10T12:48:05.353725998Z INFO: Getting status/config from prometheus-k8s-0 2026-09-10T12:48:05.479687932Z Wrote inspect data to must-gather. 2026-09-10T12:48:05.492030482Z error: the server doesn't have a resource type "kubeletconfigs" 2026-09-10T12:48:05.692906516Z Wrote inspect data to must-gather. 2026-09-10T12:48:05.727162061Z INFO: Getting status/flags from prometheus-k8s-0 2026-09-10T12:48:05.736541719Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-09-10T12:48:05.741302449Z INFO: INTERCONNECT MODE 2026-09-10T12:48:05.741335633Z INFO: Gathering ovn-kubernetes DBs 2026-09-10T12:48:05.896885345Z Wrote inspect data to must-gather. 2026-09-10T12:48:05.900168606Z INFO: Gathering OVN_Northbound from ovnkube-node-4qqqh... 2026-09-10T12:48:05.900432286Z INFO: Gathering OVN_Northbound from ovnkube-node-8rcnt... 2026-09-10T12:48:05.900666237Z INFO: Gathering OVN_Northbound from ovnkube-node-tfcmz... 2026-09-10T12:48:05.900979821Z INFO: Gathering OVN_Southbound from ovnkube-node-4qqqh... 2026-09-10T12:48:05.901299823Z INFO: Gathering OVN_Southbound from ovnkube-node-8rcnt... 2026-09-10T12:48:05.901640320Z INFO: Gathering OVN_Southbound from ovnkube-node-tfcmz... 2026-09-10T12:48:05.942380058Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:05.978720370Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-09-10T12:48:06.195980252Z tar: Removing leading `/' from member names 2026-09-10T12:48:06.201792558Z tar: Removing leading `/' from member names 2026-09-10T12:48:06.202067224Z tar: Removing leading `/' from member names 2026-09-10T12:48:06.209081013Z tar: Removing leading `/' from member names 2026-09-10T12:48:06.226557046Z tar: Removing leading `/' from member names 2026-09-10T12:48:06.251621445Z tar: Removing leading `/' from member names 2026-09-10T12:48:06.265232152Z Wrote inspect data to must-gather. 2026-09-10T12:48:06.307157653Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:06.310656688Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-09-10T12:48:06.316170230Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-09-10T12:48:06.428674851Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-09-10T12:48:06.507814625Z Wrote inspect data to must-gather. 2026-09-10T12:48:06.537250901Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:06.572781452Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-09-10T12:48:06.754192060Z Wrote inspect data to must-gather. 2026-09-10T12:48:06.770506211Z INFO: Waiting for network log collection to complete ... 2026-09-10T12:48:06.770534994Z INFO: Waiting for ovnk database copies to complete ... 2026-09-10T12:48:06.770921188Z INFO: Copying ovnk databases complete. 2026-09-10T12:48:06.773092787Z 3.5M must-gather/network_logs/ovnk_database_store 2026-09-10T12:48:06.773331423Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:06.775462384Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-09-10T12:48:06.775589340Z ovnk_database_store/ 2026-09-10T12:48:06.775589340Z ovnk_database_store/ovnkube-node-4qqqh_nbdb 2026-09-10T12:48:06.784204853Z ovnk_database_store/ovnkube-node-tfcmz_nbdb 2026-09-10T12:48:06.791870485Z ovnk_database_store/ovnkube-node-tfcmz_sbdb 2026-09-10T12:48:06.814733203Z ovnk_database_store/ovnkube-node-4qqqh_sbdb 2026-09-10T12:48:06.835448333Z ovnk_database_store/ovnkube-node-8rcnt_sbdb 2026-09-10T12:48:06.853867726Z ovnk_database_store/ovnkube-node-8rcnt_nbdb 2026-09-10T12:48:06.866412631Z INFO: Network log collection complete. 2026-09-10T12:48:06.963074876Z W0910 12:48:06.963025 1514 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-09-10T12:48:06.967559903Z Wrote inspect data to must-gather. 2026-09-10T12:48:06.967783986Z INFO: Getting status from alertmanager-main-0 2026-09-10T12:48:06.975004976Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:07.064651237Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-09-10T12:48:07.197250465Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:07.200402501Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-09-10T12:48:07.243829349Z Wrote inspect data to must-gather. 2026-09-10T12:48:07.248326466Z Gathering data for ns/openshift-network-console... 2026-09-10T12:48:07.418965958Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:07.489700185Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-09-10T12:48:07.492613405Z INFO: Fallback to identify the container image from release info 2026-09-10T12:48:07.625278678Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:07.628432870Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-09-10T12:48:07.666949237Z Gathering data for ns/openshift-console-operator... 2026-09-10T12:48:07.784333919Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:07.959556401Z Wrote inspect data to /must-gather/istio. 2026-09-10T12:48:08.017271164Z Gathering data for ns/openshift-console... 2026-09-10T12:48:08.342427582Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399 2026-09-10T12:48:08.465898871Z daemonset.apps/perf-node-gather-daemonset created 2026-09-10T12:48:08.564559228Z Gathering data for ns/openshift-cluster-storage-operator... 2026-09-10T12:48:08.596886646Z Waiting for performance profile collector pods to become ready: 1 2026-09-10T12:48:08.633347151Z Done executing Istio gather script 2026-09-10T12:48:09.023579860Z Gathering data for ns/openshift-dns-operator... 2026-09-10T12:48:09.337000918Z Gathering data for ns/openshift-dns... 2026-09-10T12:48:09.695605406Z Waiting for performance profile collector pods to become ready: 2 2026-09-10T12:48:09.722041516Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-09-10T12:48:09.884899506Z Gathering data for ns/openshift-image-registry... 2026-09-10T12:48:10.308238098Z Gathering data for ns/openshift-ingress-operator... 2026-09-10T12:48:10.599184405Z Gathering data for ns/openshift-ingress... 2026-09-10T12:48:10.802421038Z Waiting for performance profile collector pods to become ready: 3 2026-09-10T12:48:10.896820714Z Gathering data for ns/openshift-ingress-canary... 2026-09-10T12:48:11.241604383Z Gathering data for ns/openshift-insights... 2026-09-10T12:48:11.904467944Z Waiting for performance profile collector pods to become ready: 4 2026-09-10T12:48:13.003773670Z Waiting for performance profile collector pods to become ready: 5 2026-09-10T12:48:13.159657264Z Gathering data for ns/openshift-config... 2026-09-10T12:48:13.389946860Z Gathering data for ns/openshift-config-managed... 2026-09-10T12:48:13.772473861Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-09-10T12:48:14.026745945Z Gathering data for ns/openshift-kube-apiserver... 2026-09-10T12:48:14.123583659Z Waiting for performance profile collector pods to become ready: 6 2026-09-10T12:48:14.272780739Z Gathering data for ns/openshift-kube-controller-manager... 2026-09-10T12:48:14.528384734Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-09-10T12:48:14.727815612Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-09-10T12:48:14.785163428Z Gathering data for ns/openshift-kube-scheduler... 2026-09-10T12:48:15.011231760Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-09-10T12:48:15.247627086Z Waiting for performance profile collector pods to become ready: 7 2026-09-10T12:48:15.283483354Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-09-10T12:48:15.576080202Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-09-10T12:48:15.844942036Z Gathering data for ns/openshift-user-workload-monitoring... 2026-09-10T12:48:16.274166765Z Gathering data for ns/openshift-multus... 2026-09-10T12:48:16.352090360Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-09-10T12:48:16.557736423Z Collecting performance related data for node ip-10-0-143-110.ec2.internal 2026-09-10T12:48:16.564308247Z Collecting performance related data for node ip-10-0-130-221.ec2.internal 2026-09-10T12:48:16.571644672Z Collecting performance related data for node ip-10-0-136-197.ec2.internal 2026-09-10T12:48:17.732363287Z Gathering data for ns/openshift-ovn-kubernetes... 2026-09-10T12:48:18.921450860Z Gathering data for ns/openshift-host-network... 2026-09-10T12:48:19.234746696Z Gathering data for ns/openshift-network-diagnostics... 2026-09-10T12:48:19.726375449Z Gathering data for ns/openshift-network-node-identity... 2026-09-10T12:48:19.738102168Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-09-10T12:48:20.093099648Z Gathering data for ns/openshift-network-operator... 2026-09-10T12:48:20.465340951Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-09-10T12:48:20.716169701Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-09-10T12:48:21.170030615Z Gathering data for ns/openshift-apiserver-operator... 2026-09-10T12:48:21.460390697Z Gathering data for ns/openshift-apiserver... 2026-09-10T12:48:21.823644819Z Gathering data for ns/openshift-controller-manager-operator... 2026-09-10T12:48:22.066183312Z Gathering data for ns/openshift-controller-manager... 2026-09-10T12:48:22.307349679Z Gathering data for ns/openshift-cluster-samples-operator... 2026-09-10T12:48:22.903894176Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-09-10T12:48:23.166984991Z Gathering data for ns/openshift-service-ca-operator... 2026-09-10T12:48:23.450039348Z Gathering data for ns/openshift-service-ca... 2026-09-10T12:48:23.786994360Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-09-10T12:48:24.372802278Z Collecting kubelet logs for node ip-10-0-143-110.ec2.internal 2026-09-10T12:48:24.374700518Z Collecting kubelet logs for node ip-10-0-130-221.ec2.internal 2026-09-10T12:48:24.376450822Z Collecting kubelet logs for node ip-10-0-136-197.ec2.internal 2026-09-10T12:48:24.673655338Z Wrote inspect data to must-gather. 2026-09-10T12:48:24.673703672Z error: inspection completed with the errors occurred while gathering data: 2026-09-10T12:48:24.673703672Z skipping gathering secrets/support due to error: secrets "support" not found 2026-09-10T12:48:24.703177706Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-dxwh5 namespace 2026-09-10T12:48:24.708313497Z INFO: Node performance data collection complete. 2026-09-10T12:48:24.743348778Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-09-10T12:48:24.829535504Z error: the server doesn't have a resource type "clusters" 2026-09-10T12:48:24.948144170Z Caches written to disk