2026-07-28T05:56:35.377190800Z [disk usage checker] Started 2026-07-28T05:56:35.385263792Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-28T05:56:35.569068679Z Gathering data for ns/openshift-cluster-version... 2026-07-28T05:56:35.688318107Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-28T05:56:35.761361181Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-28T05:56:35.832452442Z Gathering data for ns/default... 2026-07-28T05:56:36.083366014Z Gathering data for ns/openshift... 2026-07-28T05:56:36.435820666Z Gathering data for ns/kube-system... 2026-07-28T05:56:37.096504652Z Gathering data for ns/openshift-etcd... 2026-07-28T05:56:37.386699545Z Wrote inspect data to must-gather. 2026-07-28T05:56:37.386730519Z error: inspection completed with the errors occurred while gathering data: 2026-07-28T05:56:37.386730519Z namespaces "assisted-installer" not found 2026-07-28T05:56:37.768531183Z Waiting on subprocesses to finish execution. 2026-07-28T05:56:37.774487294Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-28T05:56:37.775052698Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-28T05:56:37.786622777Z INFO: Collecting host service logs for crio 2026-07-28T05:56:37.786947646Z INFO: Collecting host service logs for kubelet 2026-07-28T05:56:37.787308335Z INFO: Collecting host service logs for rpm-ostreed 2026-07-28T05:56:37.787749921Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-28T05:56:37.788486905Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-28T05:56:37.788830614Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-28T05:56:37.789140375Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-28T05:56:37.789492657Z INFO: Collecting host service logs for NetworkManager 2026-07-28T05:56:37.789902147Z INFO: Collecting host service logs for openvswitch 2026-07-28T05:56:37.790219462Z INFO: Collecting host service logs for ovs-configuration 2026-07-28T05:56:37.790542366Z INFO: Collecting host service logs for ovsdb-server 2026-07-28T05:56:37.790611529Z Executing Istio gather script 2026-07-28T05:56:37.790906489Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-28T05:56:37.791308109Z INFO: Waiting for worker host service log collection to complete ... 2026-07-28T05:56:37.792623492Z INFO: Gathering HAProxy config files 2026-07-28T05:56:38.070462025Z INFO: Waiting for node performance related collection to complete ... 2026-07-28T05:56:38.530857988Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-28T05:56:38.729409310Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-28T05:56:38.970887469Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-28T05:56:39.008352304Z No resources found 2026-07-28T05:56:39.263628307Z INFO: "metallb-operator" not detected. Skipping. 2026-07-28T05:56:39.290872663Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-28T05:56:39.330512809Z error: the server doesn't have a resource type "performanceprofile" 2026-07-28T05:56:39.335471974Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-cvdl2 2026-07-28T05:56:39.335471974Z insights-runtime-extractor-9tcqp 2026-07-28T05:56:39.335471974Z insights-runtime-extractor-gnjsr 2026-07-28T05:56:39.335471974Z insights-runtime-extractor-v2fn6 2026-07-28T05:56:39.396608341Z Wrote inspect data to must-gather. 2026-07-28T05:56:39.427993495Z Wrote inspect data to must-gather. 2026-07-28T05:56:39.448539666Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-28T05:56:39.448617290Z INFO: on-disk MachineConfig config collection complete. 2026-07-28T05:56:39.500516681Z ERROR: No running kube-apiserver pods found 2026-07-28T05:56:39.503449224Z INFO: Worker host service log collection to complete. 2026-07-28T05:56:39.517052229Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-28T05:56:39.601267299Z No resources found 2026-07-28T05:56:39.658444643Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-28T05:56:39.741317756Z No resources found in openshift-etcd namespace. 2026-07-28T05:56:39.785256500Z Wrote inspect data to must-gather. 2026-07-28T05:56:39.901035400Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-28T05:56:39.930172151Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-28T05:56:39.947767898Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-28T05:56:39.947767898Z See 'oc rsync -h' for help and examples 2026-07-28T05:56:39.949246227Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.054826065Z Gathering data for ns/openshift-monitoring... 2026-07-28T05:56:40.067869495Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.155819761Z tar: Removing leading `/' from member names 2026-07-28T05:56:40.161741880Z INFO: HAProxy config collection complete. 2026-07-28T05:56:40.239931334Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.258292928Z INFO: 'previous-logs' folder not found on ip-10-0-137-238.ec2.internal, skipping... 2026-07-28T05:56:40.272068764Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.290919394Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-28T05:56:40.327564228Z error: the server doesn't have a resource type "machineconfigs" 2026-07-28T05:56:40.368994747Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.400852029Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-28T05:56:40.496731496Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-28T05:56:40.532306685Z INFO: 'previous-logs' folder not found on ip-10-0-141-124.ec2.internal, skipping... 2026-07-28T05:56:40.534037476Z INFO: Getting rules from prometheus-k8s-0 2026-07-28T05:56:40.636532724Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.713508502Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.761369503Z INFO: 'previous-logs' folder not found on ip-10-0-142-234.ec2.internal, skipping... 2026-07-28T05:56:40.761399966Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-28T05:56:40.761407070Z INFO: Machine Config Daemon termination log collection complete. 2026-07-28T05:56:40.859055744Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-28T05:56:40.874327975Z Wrote inspect data to must-gather. 2026-07-28T05:56:40.965743557Z INFO: Getting status/config from prometheus-k8s-0 2026-07-28T05:56:41.059389447Z Wrote inspect data to must-gather. 2026-07-28T05:56:41.136543657Z INFO: INTERCONNECT MODE 2026-07-28T05:56:41.136570522Z INFO: Gathering ovn-kubernetes DBs 2026-07-28T05:56:41.246817078Z Wrote inspect data to must-gather. 2026-07-28T05:56:41.282498175Z INFO: Gathering OVN_Northbound from ovnkube-node-8ps4k... 2026-07-28T05:56:41.282710636Z INFO: Gathering OVN_Northbound from ovnkube-node-gpmnb... 2026-07-28T05:56:41.282910483Z INFO: Gathering OVN_Northbound from ovnkube-node-m7v24... 2026-07-28T05:56:41.283181328Z INFO: Gathering OVN_Southbound from ovnkube-node-8ps4k... 2026-07-28T05:56:41.283447585Z INFO: Gathering OVN_Southbound from ovnkube-node-gpmnb... 2026-07-28T05:56:41.283741746Z INFO: Gathering OVN_Southbound from ovnkube-node-m7v24... 2026-07-28T05:56:41.430363518Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-28T05:56:41.488306844Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-28T05:56:41.564799791Z Wrote inspect data to must-gather. 2026-07-28T05:56:41.585679820Z tar: Removing leading `/' from member names 2026-07-28T05:56:41.587815882Z tar: Removing leading `/' from member names 2026-07-28T05:56:41.591236223Z tar: Removing leading `/' from member names 2026-07-28T05:56:41.626801621Z tar: Removing leading `/' from member names 2026-07-28T05:56:41.644244515Z tar: Removing leading `/' from member names 2026-07-28T05:56:41.644486061Z tar: Removing leading `/' from member names 2026-07-28T05:56:41.688934912Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:41.758970900Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-28T05:56:41.807796495Z Wrote inspect data to must-gather. 2026-07-28T05:56:41.891905608Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-28T05:56:41.931058773Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:41.934285170Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-28T05:56:42.012585460Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-28T05:56:42.025269362Z Wrote inspect data to must-gather. 2026-07-28T05:56:42.088899871Z INFO: Waiting for network log collection to complete ... 2026-07-28T05:56:42.089012893Z INFO: Waiting for ovnk database copies to complete ... 2026-07-28T05:56:42.089508847Z INFO: Copying ovnk databases complete. 2026-07-28T05:56:42.091586334Z 3.6M must-gather/network_logs/ovnk_database_store 2026-07-28T05:56:42.094498201Z ovnk_database_store/ 2026-07-28T05:56:42.094542769Z ovnk_database_store/ovnkube-node-8ps4k_nbdb 2026-07-28T05:56:42.107373380Z ovnk_database_store/ovnkube-node-gpmnb_nbdb 2026-07-28T05:56:42.117155475Z ovnk_database_store/ovnkube-node-gpmnb_sbdb 2026-07-28T05:56:42.142749332Z ovnk_database_store/ovnkube-node-8ps4k_sbdb 2026-07-28T05:56:42.155352340Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:42.168084166Z ovnk_database_store/ovnkube-node-m7v24_sbdb 2026-07-28T05:56:42.188324910Z ovnk_database_store/ovnkube-node-m7v24_nbdb 2026-07-28T05:56:42.197884880Z INFO: Network log collection complete. 2026-07-28T05:56:42.242412199Z Wrote inspect data to must-gather. 2026-07-28T05:56:42.262652287Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-28T05:56:42.365483360Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:42.369450172Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-28T05:56:42.439477814Z W0728 05:56:42.439429 1530 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-28T05:56:42.446681355Z Wrote inspect data to must-gather. 2026-07-28T05:56:42.488513562Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-28T05:56:42.568255344Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:42.638693414Z INFO: Getting status from alertmanager-main-0 2026-07-28T05:56:42.664514326Z Gathering data for ns/openshift-network-console... 2026-07-28T05:56:42.720875138Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-28T05:56:42.723729331Z INFO: Fallback to identify the container image from release info 2026-07-28T05:56:42.787809219Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:42.790657764Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-28T05:56:42.986333250Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:43.052432144Z Gathering data for ns/openshift-console-operator... 2026-07-28T05:56:43.163674878Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:43.165914060Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-28T05:56:43.308606373Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:43.321523844Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811 2026-07-28T05:56:43.426928459Z Gathering data for ns/openshift-console... 2026-07-28T05:56:43.443664007Z daemonset.apps/perf-node-gather-daemonset created 2026-07-28T05:56:43.499469503Z Wrote inspect data to /must-gather/istio. 2026-07-28T05:56:43.577987768Z Waiting for performance profile collector pods to become ready: 1 2026-07-28T05:56:43.929996363Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-28T05:56:44.125456324Z Done executing Istio gather script 2026-07-28T05:56:44.366084390Z Gathering data for ns/openshift-dns-operator... 2026-07-28T05:56:44.600755280Z Gathering data for ns/openshift-dns... 2026-07-28T05:56:44.676064846Z Waiting for performance profile collector pods to become ready: 2 2026-07-28T05:56:45.119045496Z Gathering data for ns/openshift-image-registry... 2026-07-28T05:56:45.406652244Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-28T05:56:45.520525358Z Gathering data for ns/openshift-ingress-operator... 2026-07-28T05:56:45.791253518Z Waiting for performance profile collector pods to become ready: 3 2026-07-28T05:56:45.801652817Z Gathering data for ns/openshift-ingress... 2026-07-28T05:56:46.068157071Z Gathering data for ns/openshift-ingress-canary... 2026-07-28T05:56:46.391572461Z Gathering data for ns/openshift-insights... 2026-07-28T05:56:46.903056029Z Waiting for performance profile collector pods to become ready: 4 2026-07-28T05:56:48.002925125Z Waiting for performance profile collector pods to become ready: 5 2026-07-28T05:56:48.371201914Z Gathering data for ns/openshift-config... 2026-07-28T05:56:48.601714924Z Gathering data for ns/openshift-config-managed... 2026-07-28T05:56:48.967185905Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-28T05:56:49.107356529Z Waiting for performance profile collector pods to become ready: 6 2026-07-28T05:56:49.187702310Z Gathering data for ns/openshift-kube-apiserver... 2026-07-28T05:56:49.393839828Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-28T05:56:49.638744365Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-28T05:56:49.868721481Z Gathering data for ns/openshift-kube-scheduler... 2026-07-28T05:56:50.083289832Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-28T05:56:50.214213174Z Waiting for performance profile collector pods to become ready: 7 2026-07-28T05:56:50.366559227Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-28T05:56:50.411459148Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-28T05:56:50.652099233Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-28T05:56:50.929663690Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-28T05:56:51.329176659Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-28T05:56:51.360258236Z Gathering data for ns/openshift-multus... 2026-07-28T05:56:51.546069571Z Collecting performance related data for node ip-10-0-141-124.ec2.internal 2026-07-28T05:56:51.552556548Z Collecting performance related data for node ip-10-0-142-234.ec2.internal 2026-07-28T05:56:51.559367999Z Collecting performance related data for node ip-10-0-137-238.ec2.internal 2026-07-28T05:56:52.816771577Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-28T05:56:53.994828596Z Gathering data for ns/openshift-host-network... 2026-07-28T05:56:54.278449619Z Gathering data for ns/openshift-network-diagnostics... 2026-07-28T05:56:54.753215692Z Gathering data for ns/openshift-network-node-identity... 2026-07-28T05:56:55.093401234Z Gathering data for ns/openshift-network-operator... 2026-07-28T05:56:55.416835267Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-28T05:56:55.423455456Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-28T05:56:55.652311066Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-28T05:56:56.073840951Z Gathering data for ns/openshift-apiserver-operator... 2026-07-28T05:56:56.326153871Z Gathering data for ns/openshift-apiserver... 2026-07-28T05:56:56.713200911Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-28T05:56:57.000627284Z Gathering data for ns/openshift-controller-manager... 2026-07-28T05:56:57.226595322Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-28T05:56:57.771298527Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-28T05:56:58.003351176Z Gathering data for ns/openshift-service-ca-operator... 2026-07-28T05:56:58.284543172Z Gathering data for ns/openshift-service-ca... 2026-07-28T05:56:58.627281054Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-28T05:56:59.009712768Z Collecting kubelet logs for node ip-10-0-141-124.ec2.internal 2026-07-28T05:56:59.011481449Z Collecting kubelet logs for node ip-10-0-142-234.ec2.internal 2026-07-28T05:56:59.013234904Z Collecting kubelet logs for node ip-10-0-137-238.ec2.internal 2026-07-28T05:56:59.349607897Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-26j78 namespace 2026-07-28T05:56:59.354607319Z INFO: Node performance data collection complete. 2026-07-28T05:56:59.510469475Z Wrote inspect data to must-gather. 2026-07-28T05:56:59.510512234Z error: inspection completed with the errors occurred while gathering data: 2026-07-28T05:56:59.510512234Z skipping gathering secrets/support due to error: secrets "support" not found 2026-07-28T05:56:59.638656233Z error: the server doesn't have a resource type "clusters" 2026-07-28T05:56:59.769498326Z Caches written to disk