2026-07-09T15:41:19.381817388Z [disk usage checker] Started 2026-07-09T15:41:19.386328409Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-09T15:41:19.595113323Z Gathering data for ns/openshift-cluster-version... 2026-07-09T15:41:19.736813500Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-09T15:41:19.825339730Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-09T15:41:19.918177031Z Gathering data for ns/default... 2026-07-09T15:41:20.244871783Z Gathering data for ns/openshift... 2026-07-09T15:41:20.700444381Z Gathering data for ns/kube-system... 2026-07-09T15:41:21.383053335Z Gathering data for ns/openshift-etcd... 2026-07-09T15:41:21.692469247Z Wrote inspect data to must-gather. 2026-07-09T15:41:21.692507091Z error: inspection completed with the errors occurred while gathering data: 2026-07-09T15:41:21.692507091Z namespaces "assisted-installer" not found 2026-07-09T15:41:22.202458862Z Waiting on subprocesses to finish execution. 2026-07-09T15:41:22.206663289Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-09T15:41:22.209121285Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-09T15:41:22.220402857Z Executing Istio gather script 2026-07-09T15:41:22.229793284Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-09T15:41:22.237717359Z INFO: Gathering HAProxy config files 2026-07-09T15:41:22.247316013Z INFO: Waiting for node performance related collection to complete ... 2026-07-09T15:41:22.296164557Z INFO: Collecting host service logs for crio 2026-07-09T15:41:22.296441773Z INFO: Collecting host service logs for kubelet 2026-07-09T15:41:22.296764633Z INFO: Collecting host service logs for rpm-ostreed 2026-07-09T15:41:22.297040835Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-09T15:41:22.297301035Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-09T15:41:22.297587170Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-09T15:41:22.297880434Z INFO: Collecting host service logs for NetworkManager 2026-07-09T15:41:22.298155002Z INFO: Collecting host service logs for openvswitch 2026-07-09T15:41:22.298430110Z INFO: Collecting host service logs for ovs-configuration 2026-07-09T15:41:22.298779020Z INFO: Collecting host service logs for ovsdb-server 2026-07-09T15:41:22.299090009Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-09T15:41:22.299478689Z INFO: Waiting for worker host service log collection to complete ... 2026-07-09T15:41:22.872735471Z INFO: "metallb-operator" not detected. Skipping. 2026-07-09T15:41:22.976291466Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-09T15:41:22.985159726Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-szv5l 2026-07-09T15:41:22.985231289Z insights-runtime-extractor-9ng75 2026-07-09T15:41:22.985253041Z insights-runtime-extractor-d7thv 2026-07-09T15:41:22.985272321Z insights-runtime-extractor-fvdf6 2026-07-09T15:41:23.065469138Z Wrote inspect data to must-gather. 2026-07-09T15:41:23.183661204Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-09T15:41:23.327135416Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-09T15:41:23.327174345Z INFO: on-disk MachineConfig config collection complete. 2026-07-09T15:41:23.352838502Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-09T15:41:23.418487240Z No resources found in openshift-etcd namespace. 2026-07-09T15:41:23.515142883Z error: the server doesn't have a resource type "performanceprofile" 2026-07-09T15:41:23.765737259Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-09T15:41:23.765737259Z See 'oc rsync -h' for help and examples 2026-07-09T15:41:23.772213458Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-09T15:41:23.822648779Z Wrote inspect data to must-gather. 2026-07-09T15:41:23.895258549Z No resources found 2026-07-09T15:41:24.180423865Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-09T15:41:24.440030587Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-09T15:41:24.509258130Z Wrote inspect data to must-gather. 2026-07-09T15:41:24.554030405Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-09T15:41:24.619477388Z INFO: Worker host service log collection to complete. 2026-07-09T15:41:24.694895507Z INFO: 'previous-logs' folder not found on ip-10-0-128-26.ec2.internal, skipping... 2026-07-09T15:41:24.702833151Z Wrote inspect data to must-gather. 2026-07-09T15:41:24.738471359Z Wrote inspect data to must-gather. 2026-07-09T15:41:24.782362814Z Wrote inspect data to must-gather. 2026-07-09T15:41:24.847347496Z ERROR: No running kube-apiserver pods found 2026-07-09T15:41:24.847991093Z No resources found 2026-07-09T15:41:24.894928171Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-09T15:41:24.911172353Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-09T15:41:25.014943828Z Wrote inspect data to must-gather. 2026-07-09T15:41:25.096288274Z Gathering data for ns/openshift-monitoring... 2026-07-09T15:41:25.137298023Z INFO: 'previous-logs' folder not found on ip-10-0-136-153.ec2.internal, skipping... 2026-07-09T15:41:25.169159318Z tar: Removing leading `/' from member names 2026-07-09T15:41:25.174460812Z INFO: HAProxy config collection complete. 2026-07-09T15:41:25.222683745Z Wrote inspect data to must-gather. 2026-07-09T15:41:25.263588177Z error: the server doesn't have a resource type "machineconfigs" 2026-07-09T15:41:25.463827121Z INFO: 'previous-logs' folder not found on ip-10-0-137-233.ec2.internal, skipping... 2026-07-09T15:41:25.463920159Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-09T15:41:25.463979422Z INFO: Machine Config Daemon termination log collection complete. 2026-07-09T15:41:25.467582178Z INFO: Getting rules from prometheus-k8s-0 2026-07-09T15:41:25.499993013Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-09T15:41:25.566769819Z Wrote inspect data to must-gather. 2026-07-09T15:41:25.771659275Z Wrote inspect data to must-gather. 2026-07-09T15:41:25.908787494Z Wrote inspect data to must-gather. 2026-07-09T15:41:25.976226107Z INFO: Getting status/config from prometheus-k8s-0 2026-07-09T15:41:25.981151639Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-09T15:41:25.996433621Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-09T15:41:26.203775727Z Wrote inspect data to must-gather. 2026-07-09T15:41:26.256039197Z INFO: INTERCONNECT MODE 2026-07-09T15:41:26.256062648Z INFO: Gathering ovn-kubernetes DBs 2026-07-09T15:41:26.421222909Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-09T15:41:26.431863641Z INFO: Gathering OVN_Northbound from ovnkube-node-f7mcr... 2026-07-09T15:41:26.432105154Z INFO: Gathering OVN_Northbound from ovnkube-node-pmqf5... 2026-07-09T15:41:26.432360160Z INFO: Gathering OVN_Northbound from ovnkube-node-q9v9p... 2026-07-09T15:41:26.432692251Z INFO: Gathering OVN_Southbound from ovnkube-node-f7mcr... 2026-07-09T15:41:26.433158616Z INFO: Gathering OVN_Southbound from ovnkube-node-pmqf5... 2026-07-09T15:41:26.433441256Z INFO: Gathering OVN_Southbound from ovnkube-node-q9v9p... 2026-07-09T15:41:26.453654281Z Wrote inspect data to must-gather. 2026-07-09T15:41:26.726784998Z tar: Removing leading `/' from member names 2026-07-09T15:41:26.742601063Z tar: Removing leading `/' from member names 2026-07-09T15:41:26.774431312Z tar: Removing leading `/' from member names 2026-07-09T15:41:26.784965767Z tar: Removing leading `/' from member names 2026-07-09T15:41:26.790470137Z tar: Removing leading `/' from member names 2026-07-09T15:41:26.842710916Z tar: Removing leading `/' from member names 2026-07-09T15:41:26.862643511Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-09T15:41:26.886201484Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-09T15:41:26.893212120Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-09T15:41:26.905378059Z Wrote inspect data to must-gather. 2026-07-09T15:41:27.091289986Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:27.182899588Z Wrote inspect data to must-gather. 2026-07-09T15:41:27.184044847Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-09T15:41:27.353500925Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:27.359187575Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-09T15:41:27.362268663Z INFO: Waiting for network log collection to complete ... 2026-07-09T15:41:27.362317767Z INFO: Waiting for ovnk database copies to complete ... 2026-07-09T15:41:27.362755111Z INFO: Copying ovnk databases complete. 2026-07-09T15:41:27.364347075Z 2.9M must-gather/network_logs/ovnk_database_store 2026-07-09T15:41:27.366873423Z ovnk_database_store/ 2026-07-09T15:41:27.366873423Z ovnk_database_store/ovnkube-node-q9v9p_nbdb 2026-07-09T15:41:27.377386033Z ovnk_database_store/ovnkube-node-pmqf5_nbdb 2026-07-09T15:41:27.384903163Z ovnk_database_store/ovnkube-node-f7mcr_sbdb 2026-07-09T15:41:27.405065841Z ovnk_database_store/ovnkube-node-pmqf5_sbdb 2026-07-09T15:41:27.422836290Z ovnk_database_store/ovnkube-node-q9v9p_sbdb 2026-07-09T15:41:27.437374999Z Wrote inspect data to must-gather. 2026-07-09T15:41:27.438874319Z ovnk_database_store/ovnkube-node-f7mcr_nbdb 2026-07-09T15:41:27.449567717Z INFO: Network log collection complete. 2026-07-09T15:41:27.475137130Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-09T15:41:27.581200998Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:27.587490430Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-09T15:41:27.745250181Z Wrote inspect data to must-gather. 2026-07-09T15:41:27.794607070Z Gathering data for ns/openshift-network-console... 2026-07-09T15:41:27.822954079Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:27.825695339Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-09T15:41:27.925649112Z INFO: Getting status from alertmanager-main-0 2026-07-09T15:41:28.070822557Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:28.077443724Z Wrote inspect data to must-gather. 2026-07-09T15:41:28.296176176Z Gathering data for ns/openshift-console-operator... 2026-07-09T15:41:28.301073991Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:28.303717276Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-09T15:41:28.375013193Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-09T15:41:28.377378875Z INFO: Fallback to identify the container image from release info 2026-07-09T15:41:28.535786167Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:28.713509883Z Gathering data for ns/openshift-console... 2026-07-09T15:41:28.733831295Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:28.736197297Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-09T15:41:28.913655226Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:29.109554845Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:29.112943748Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-09T15:41:29.245295889Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-09T15:41:29.290381556Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:29.482032528Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-09T15:41:29.499357757Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:29.501649810Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-09T15:41:29.676972480Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:29.776214702Z Gathering data for ns/openshift-dns-operator... 2026-07-09T15:41:29.892888775Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:29.895231369Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-09T15:41:30.073601055Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:30.127412514Z Gathering data for ns/openshift-dns... 2026-07-09T15:41:30.259847030Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:30.262437651Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-09T15:41:30.423371251Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:30.628152496Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:30.631169499Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-09T15:41:30.803254298Z Gathering data for ns/openshift-image-registry... 2026-07-09T15:41:30.829364822Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d 2026-07-09T15:41:30.835256103Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:30.964659454Z daemonset.apps/perf-node-gather-daemonset created 2026-07-09T15:41:31.063347780Z Wrote inspect data to /must-gather/istio. 2026-07-09T15:41:31.115900522Z Waiting for performance profile collector pods to become ready: 1 2026-07-09T15:41:31.383332163Z Gathering data for ns/openshift-ingress-operator... 2026-07-09T15:41:31.798137022Z Gathering data for ns/openshift-ingress... 2026-07-09T15:41:31.838697112Z Done executing Istio gather script 2026-07-09T15:41:32.148180717Z Gathering data for ns/openshift-ingress-canary... 2026-07-09T15:41:32.237897497Z Waiting for performance profile collector pods to become ready: 2 2026-07-09T15:41:32.557998914Z Gathering data for ns/openshift-insights... 2026-07-09T15:41:33.360767342Z Waiting for performance profile collector pods to become ready: 3 2026-07-09T15:41:34.477686172Z Waiting for performance profile collector pods to become ready: 4 2026-07-09T15:41:34.486669406Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-09T15:41:34.848612991Z Gathering data for ns/kserve... 2026-07-09T15:41:35.588247987Z Waiting for performance profile collector pods to become ready: 5 2026-07-09T15:41:36.118375615Z Gathering data for ns/openshift-config... 2026-07-09T15:41:36.389667111Z Gathering data for ns/openshift-config-managed... 2026-07-09T15:41:36.710742409Z Waiting for performance profile collector pods to become ready: 6 2026-07-09T15:41:36.824612455Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-09T15:41:37.078379629Z Gathering data for ns/openshift-kube-apiserver... 2026-07-09T15:41:37.342237277Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-09T15:41:37.620151604Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-09T15:41:37.853667313Z Waiting for performance profile collector pods to become ready: 7 2026-07-09T15:41:37.916080149Z Gathering data for ns/openshift-kube-scheduler... 2026-07-09T15:41:38.208356895Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-09T15:41:38.493308291Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-09T15:41:38.806578081Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-09T15:41:38.990632512Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-09T15:41:39.129637080Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-09T15:41:39.265416439Z Collecting performance related data for node ip-10-0-137-233.ec2.internal 2026-07-09T15:41:39.275361477Z Collecting performance related data for node ip-10-0-136-153.ec2.internal 2026-07-09T15:41:39.285124227Z Collecting performance related data for node ip-10-0-128-26.ec2.internal 2026-07-09T15:41:39.492484097Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-09T15:41:39.934558718Z Gathering data for ns/openshift-multus... 2026-07-09T15:41:41.524789909Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-09T15:41:42.834859498Z Gathering data for ns/openshift-host-network... 2026-07-09T15:41:43.143178829Z Gathering data for ns/openshift-network-diagnostics... 2026-07-09T15:41:43.695585984Z Gathering data for ns/openshift-network-node-identity... 2026-07-09T15:41:44.117397736Z Gathering data for ns/openshift-network-operator... 2026-07-09T15:41:44.501104502Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 2026-07-09T15:41:44.504305945Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-09T15:41:44.780007672Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-09T15:41:45.274942774Z Gathering data for ns/openshift-apiserver-operator... 2026-07-09T15:41:45.582762909Z Gathering data for ns/openshift-apiserver... 2026-07-09T15:41:45.985364282Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-09T15:41:46.287911359Z Gathering data for ns/openshift-controller-manager... 2026-07-09T15:41:46.575643411Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-09T15:41:47.265335223Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-09T15:41:47.574351189Z Gathering data for ns/openshift-service-ca-operator... 2026-07-09T15:41:47.761561037Z Collecting kubelet logs for node ip-10-0-137-233.ec2.internal 2026-07-09T15:41:47.764120786Z Collecting kubelet logs for node ip-10-0-136-153.ec2.internal 2026-07-09T15:41:47.766466443Z Collecting kubelet logs for node ip-10-0-128-26.ec2.internal 2026-07-09T15:41:47.936091477Z Gathering data for ns/openshift-service-ca... 2026-07-09T15:41:48.134033302Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-s2hw6 namespace 2026-07-09T15:41:48.138658433Z INFO: Node performance data collection complete. 2026-07-09T15:41:48.324878337Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-09T15:41:49.200138955Z Wrote inspect data to must-gather. 2026-07-09T15:41:49.200181006Z error: inspection completed with the errors occurred while gathering data: 2026-07-09T15:41:49.200181006Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-09T15:41:49.326075751Z error: the server doesn't have a resource type "clusters" 2026-07-09T15:41:49.471864065Z Caches written to disk