2026-07-06T15:41:28.306906592Z [disk usage checker] Started 2026-07-06T15:41:28.310664049Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-06T15:41:28.500229519Z Gathering data for ns/openshift-cluster-version... 2026-07-06T15:41:28.640065104Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-06T15:41:28.729850499Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T15:41:28.820842575Z Gathering data for ns/default... 2026-07-06T15:41:29.109069253Z Gathering data for ns/openshift... 2026-07-06T15:41:29.555749821Z Gathering data for ns/kube-system... 2026-07-06T15:41:30.263366622Z Gathering data for ns/openshift-etcd... 2026-07-06T15:41:30.553706598Z Wrote inspect data to must-gather. 2026-07-06T15:41:30.553739408Z error: inspection completed with the errors occurred while gathering data: 2026-07-06T15:41:30.553739408Z namespaces "assisted-installer" not found 2026-07-06T15:41:31.051007148Z Waiting on subprocesses to finish execution. 2026-07-06T15:41:31.059114333Z INFO: Gathering HAProxy config files 2026-07-06T15:41:31.063689201Z Executing Istio gather script 2026-07-06T15:41:31.069732318Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-06T15:41:31.070975851Z INFO: Waiting for node performance related collection to complete ... 2026-07-06T15:41:31.074219498Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-06T15:41:31.086591681Z INFO: Collecting host service logs for crio 2026-07-06T15:41:31.087017634Z INFO: Collecting host service logs for kubelet 2026-07-06T15:41:31.088989575Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-06T15:41:31.092592193Z INFO: Collecting host service logs for rpm-ostreed 2026-07-06T15:41:31.092929798Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-06T15:41:31.093290570Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-06T15:41:31.093618071Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-06T15:41:31.097627853Z INFO: Collecting host service logs for NetworkManager 2026-07-06T15:41:31.097983385Z INFO: Collecting host service logs for openvswitch 2026-07-06T15:41:31.098352702Z INFO: Collecting host service logs for ovs-configuration 2026-07-06T15:41:31.098695267Z INFO: Collecting host service logs for ovsdb-server 2026-07-06T15:41:31.099031062Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-06T15:41:31.099455402Z INFO: Waiting for worker host service log collection to complete ... 2026-07-06T15:41:31.415100504Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-06T15:41:31.646173491Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-06T15:41:31.646274916Z INFO: on-disk MachineConfig config collection complete. 2026-07-06T15:41:31.863758620Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-06T15:41:31.863994534Z No resources found in openshift-etcd namespace. 2026-07-06T15:41:31.990597922Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-w6m68 2026-07-06T15:41:31.990644778Z insights-runtime-extractor-8mlvs 2026-07-06T15:41:31.990664545Z insights-runtime-extractor-9dht5 2026-07-06T15:41:31.990684572Z insights-runtime-extractor-bbpsk 2026-07-06T15:41:32.119692232Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-06T15:41:32.154994188Z Wrote inspect data to must-gather. 2026-07-06T15:41:32.307662292Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-06T15:41:32.617837183Z INFO: "metallb-operator" not detected. Skipping. 2026-07-06T15:41:32.710534366Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T15:41:32.719796448Z error: the server doesn't have a resource type "performanceprofile" 2026-07-06T15:41:32.741613240Z No resources found 2026-07-06T15:41:32.759668622Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-06T15:41:32.816422837Z INFO: Worker host service log collection to complete. 2026-07-06T15:41:32.892503616Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-06T15:41:32.892503616Z See 'oc rsync -h' for help and examples 2026-07-06T15:41:32.972045862Z Wrote inspect data to must-gather. 2026-07-06T15:41:33.094109167Z ERROR: No running kube-apiserver pods found 2026-07-06T15:41:33.269365704Z Wrote inspect data to must-gather. 2026-07-06T15:41:33.276574775Z tar: Removing leading `/' from member names 2026-07-06T15:41:33.289767305Z INFO: HAProxy config collection complete. 2026-07-06T15:41:33.338537606Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-06T15:41:33.383655781Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-06T15:41:33.404796361Z No resources found 2026-07-06T15:41:33.486709367Z Wrote inspect data to must-gather. 2026-07-06T15:41:33.588094025Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-06T15:41:33.718226635Z Wrote inspect data to must-gather. 2026-07-06T15:41:33.729335330Z Wrote inspect data to must-gather. 2026-07-06T15:41:33.757017689Z Gathering data for ns/openshift-monitoring... 2026-07-06T15:41:33.774753403Z Wrote inspect data to must-gather. 2026-07-06T15:41:33.850074586Z INFO: 'previous-logs' folder not found on ip-10-0-136-123.ec2.internal, skipping... 2026-07-06T15:41:33.936680819Z error: the server doesn't have a resource type "machineconfigs" 2026-07-06T15:41:33.953610127Z Wrote inspect data to must-gather. 2026-07-06T15:41:34.105081610Z INFO: Getting rules from prometheus-k8s-0 2026-07-06T15:41:34.144389004Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-06T15:41:34.173507280Z INFO: 'previous-logs' folder not found on ip-10-0-137-64.ec2.internal, skipping... 2026-07-06T15:41:34.245041784Z Wrote inspect data to must-gather. 2026-07-06T15:41:34.435280759Z Wrote inspect data to must-gather. 2026-07-06T15:41:34.505745451Z INFO: 'previous-logs' folder not found on ip-10-0-138-56.ec2.internal, skipping... 2026-07-06T15:41:34.505857074Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-06T15:41:34.505907729Z INFO: Machine Config Daemon termination log collection complete. 2026-07-06T15:41:34.560849744Z Wrote inspect data to must-gather. 2026-07-06T15:41:34.602778217Z INFO: Getting status/config from prometheus-k8s-0 2026-07-06T15:41:34.612433166Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-06T15:41:34.642578512Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-06T15:41:34.866026844Z Wrote inspect data to must-gather. 2026-07-06T15:41:34.897793477Z INFO: INTERCONNECT MODE 2026-07-06T15:41:34.897831616Z INFO: Gathering ovn-kubernetes DBs 2026-07-06T15:41:35.019541694Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-06T15:41:35.099053363Z Wrote inspect data to must-gather. 2026-07-06T15:41:35.132385686Z INFO: Gathering OVN_Northbound from ovnkube-node-54gxb... 2026-07-06T15:41:35.132650820Z INFO: Gathering OVN_Northbound from ovnkube-node-8sbk2... 2026-07-06T15:41:35.132932786Z INFO: Gathering OVN_Northbound from ovnkube-node-zhgb2... 2026-07-06T15:41:35.133232506Z INFO: Gathering OVN_Southbound from ovnkube-node-54gxb... 2026-07-06T15:41:35.133526215Z INFO: Gathering OVN_Southbound from ovnkube-node-8sbk2... 2026-07-06T15:41:35.133797544Z INFO: Gathering OVN_Southbound from ovnkube-node-zhgb2... 2026-07-06T15:41:35.294374872Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-06T15:41:35.427084906Z tar: Removing leading `/' from member names 2026-07-06T15:41:35.445439733Z tar: Removing leading `/' from member names 2026-07-06T15:41:35.457152316Z Wrote inspect data to must-gather. 2026-07-06T15:41:35.485415816Z tar: Removing leading `/' from member names 2026-07-06T15:41:35.485535800Z tar: Removing leading `/' from member names 2026-07-06T15:41:35.507935964Z tar: Removing leading `/' from member names 2026-07-06T15:41:35.511321937Z tar: Removing leading `/' from member names 2026-07-06T15:41:35.592821303Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-06T15:41:35.640831522Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-06T15:41:35.722369542Z Wrote inspect data to must-gather. 2026-07-06T15:41:35.728735742Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-06T15:41:35.817966480Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:35.993231082Z Wrote inspect data to must-gather. 2026-07-06T15:41:36.022106736Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-06T15:41:36.038272728Z INFO: Waiting for network log collection to complete ... 2026-07-06T15:41:36.038306430Z INFO: Waiting for ovnk database copies to complete ... 2026-07-06T15:41:36.038787802Z INFO: Copying ovnk databases complete. 2026-07-06T15:41:36.040850854Z 3.0M must-gather/network_logs/ovnk_database_store 2026-07-06T15:41:36.043989085Z ovnk_database_store/ 2026-07-06T15:41:36.044044659Z ovnk_database_store/ovnkube-node-54gxb_nbdb 2026-07-06T15:41:36.053057182Z ovnk_database_store/ovnkube-node-54gxb_sbdb 2026-07-06T15:41:36.064531796Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:36.066926350Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-06T15:41:36.073486081Z ovnk_database_store/ovnkube-node-zhgb2_nbdb 2026-07-06T15:41:36.082916391Z ovnk_database_store/ovnkube-node-8sbk2_nbdb 2026-07-06T15:41:36.093505983Z ovnk_database_store/ovnkube-node-zhgb2_sbdb 2026-07-06T15:41:36.113308443Z ovnk_database_store/ovnkube-node-8sbk2_sbdb 2026-07-06T15:41:36.131687649Z INFO: Network log collection complete. 2026-07-06T15:41:36.231042116Z W0706 15:41:36.230987 1468 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-06T15:41:36.238602818Z Wrote inspect data to must-gather. 2026-07-06T15:41:36.288270746Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:36.342641006Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-06T15:41:36.481435963Z INFO: Getting status from alertmanager-main-0 2026-07-06T15:41:36.526800045Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:36.531195926Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-06T15:41:36.547075652Z Gathering data for ns/openshift-network-console... 2026-07-06T15:41:36.582716706Z Wrote inspect data to must-gather. 2026-07-06T15:41:36.733763624Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:36.888632532Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-06T15:41:36.891568545Z INFO: Fallback to identify the container image from release info 2026-07-06T15:41:36.961645959Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:36.965209369Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-06T15:41:37.028502320Z Gathering data for ns/openshift-console-operator... 2026-07-06T15:41:37.197781590Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:37.403109994Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:37.406535262Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-06T15:41:37.424635092Z Gathering data for ns/openshift-console... 2026-07-06T15:41:37.608187854Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:37.628918440Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a 2026-07-06T15:41:37.777225179Z daemonset.apps/perf-node-gather-daemonset created 2026-07-06T15:41:37.839360080Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:37.842528500Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-06T15:41:37.933041096Z Waiting for performance profile collector pods to become ready: 1 2026-07-06T15:41:38.022927589Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-06T15:41:38.035269352Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:38.264515452Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:38.268684259Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-06T15:41:38.358534078Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70 2026-07-06T15:41:38.472601403Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:38.591371806Z Gathering data for ns/openshift-dns-operator... 2026-07-06T15:41:38.687897392Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:38.690416320Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-06T15:41:38.858522777Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:38.939111151Z Gathering data for ns/openshift-dns... 2026-07-06T15:41:39.056922354Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:39.059318114Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-06T15:41:39.067047524Z Waiting for performance profile collector pods to become ready: 2 2026-07-06T15:41:39.217301336Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:39.409970479Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:39.412174988Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-06T15:41:39.591509365Z Gathering data for ns/openshift-image-registry... 2026-07-06T15:41:39.597489668Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:39.795476561Z Wrote inspect data to /must-gather/istio. 2026-07-06T15:41:40.110928011Z Gathering data for ns/openshift-ingress-operator... 2026-07-06T15:41:40.216870937Z Waiting for performance profile collector pods to become ready: 3 2026-07-06T15:41:40.488720577Z Gathering data for ns/openshift-ingress... 2026-07-06T15:41:40.521535531Z Done executing Istio gather script 2026-07-06T15:41:40.815759719Z Gathering data for ns/openshift-ingress-canary... 2026-07-06T15:41:41.241158987Z Gathering data for ns/openshift-insights... 2026-07-06T15:41:41.335837580Z Waiting for performance profile collector pods to become ready: 4 2026-07-06T15:41:42.450999140Z Waiting for performance profile collector pods to become ready: 5 2026-07-06T15:41:43.371009882Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T15:41:43.488505410Z Gathering data for ns/kserve... 2026-07-06T15:41:43.564884584Z Waiting for performance profile collector pods to become ready: 6 2026-07-06T15:41:44.691518506Z Waiting for performance profile collector pods to become ready: 7 2026-07-06T15:41:44.925665560Z Gathering data for ns/openshift-config... 2026-07-06T15:41:45.270879843Z Gathering data for ns/openshift-config-managed... 2026-07-06T15:41:45.662113815Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-06T15:41:45.819035316Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-06T15:41:45.943758834Z Gathering data for ns/openshift-kube-apiserver... 2026-07-06T15:41:46.101759801Z Collecting performance related data for node ip-10-0-137-64.ec2.internal 2026-07-06T15:41:46.112216984Z Collecting performance related data for node ip-10-0-138-56.ec2.internal 2026-07-06T15:41:46.124039117Z Collecting performance related data for node ip-10-0-136-123.ec2.internal 2026-07-06T15:41:46.270858640Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-06T15:41:46.619782017Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-06T15:41:46.967838713Z Gathering data for ns/openshift-kube-scheduler... 2026-07-06T15:41:47.330510665Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-06T15:41:47.695598484Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-06T15:41:48.062382928Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-06T15:41:48.372398892Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-06T15:41:48.375971249Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T15:41:48.943527778Z Gathering data for ns/openshift-multus... 2026-07-06T15:41:50.479203140Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-06T15:41:51.855550812Z Gathering data for ns/openshift-host-network... 2026-07-06T15:41:52.199525039Z Gathering data for ns/openshift-network-diagnostics... 2026-07-06T15:41:52.732714528Z Gathering data for ns/openshift-network-node-identity... 2026-07-06T15:41:53.134457630Z Gathering data for ns/openshift-network-operator... 2026-07-06T15:41:53.381055861Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T15:41:53.509372105Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-06T15:41:53.802109970Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-06T15:41:54.284299215Z Gathering data for ns/openshift-apiserver-operator... 2026-07-06T15:41:54.302210482Z Collecting kubelet logs for node ip-10-0-137-64.ec2.internal 2026-07-06T15:41:54.304430625Z Collecting kubelet logs for node ip-10-0-138-56.ec2.internal 2026-07-06T15:41:54.307521900Z Collecting kubelet logs for node ip-10-0-136-123.ec2.internal 2026-07-06T15:41:54.649596197Z Gathering data for ns/openshift-apiserver... 2026-07-06T15:41:54.680798730Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-zc8hc namespace 2026-07-06T15:41:54.686980021Z INFO: Node performance data collection complete. 2026-07-06T15:41:55.103531154Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-06T15:41:55.408572060Z Gathering data for ns/openshift-controller-manager... 2026-07-06T15:41:55.681113552Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-06T15:41:56.313224897Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-06T15:41:56.597578562Z Gathering data for ns/openshift-service-ca-operator... 2026-07-06T15:41:56.903425989Z Gathering data for ns/openshift-service-ca... 2026-07-06T15:41:57.279276414Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-06T15:41:58.173686283Z Wrote inspect data to must-gather. 2026-07-06T15:41:58.173733775Z error: inspection completed with the errors occurred while gathering data: 2026-07-06T15:41:58.173733775Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-06T15:41:58.303148487Z error: the server doesn't have a resource type "clusters" 2026-07-06T15:41:58.386714629Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-06T15:41:58.442065524Z Caches written to disk