2026-07-13T20:34:34.041991539Z [disk usage checker] Started 2026-07-13T20:34:34.045439327Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-13T20:34:34.255918255Z Gathering data for ns/openshift-cluster-version... 2026-07-13T20:34:34.401232766Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-13T20:34:34.503119135Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-13T20:34:34.594429435Z Gathering data for ns/default... 2026-07-13T20:34:34.947257531Z Gathering data for ns/openshift... 2026-07-13T20:34:35.402056658Z Gathering data for ns/kube-system... 2026-07-13T20:34:36.122166085Z Gathering data for ns/openshift-etcd... 2026-07-13T20:34:36.489072384Z Wrote inspect data to must-gather. 2026-07-13T20:34:36.489111943Z error: inspection completed with the errors occurred while gathering data: 2026-07-13T20:34:36.489111943Z namespaces "assisted-installer" not found 2026-07-13T20:34:37.051419185Z Waiting on subprocesses to finish execution. 2026-07-13T20:34:37.060383019Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-07-13T20:34:37.065051091Z INFO: Gathering machine config daemon's old logs from all nodes 2026-07-13T20:34:37.067019973Z Executing Istio gather script 2026-07-13T20:34:37.081211932Z INFO: Gathering HAProxy config files 2026-07-13T20:34:37.081324421Z INFO: Collecting host service logs for crio 2026-07-13T20:34:37.081721036Z INFO: Collecting host service logs for kubelet 2026-07-13T20:34:37.082033082Z INFO: Collecting host service logs for rpm-ostreed 2026-07-13T20:34:37.082315304Z INFO: Collecting host service logs for ostree-finalize-staged 2026-07-13T20:34:37.082630265Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-07-13T20:34:37.082938380Z INFO: Collecting host service logs for machine-config-daemon-host 2026-07-13T20:34:37.084925026Z INFO: Collecting host service logs for NetworkManager 2026-07-13T20:34:37.085379513Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-07-13T20:34:37.085841023Z INFO: Collecting host service logs for openvswitch 2026-07-13T20:34:37.090985981Z INFO: Collecting host service logs for ovs-configuration 2026-07-13T20:34:37.091517435Z INFO: Collecting host service logs for ovsdb-server 2026-07-13T20:34:37.091889174Z INFO: Collecting host service logs for ovs-vswitchd 2026-07-13T20:34:37.092309102Z INFO: Waiting for worker host service log collection to complete ... 2026-07-13T20:34:37.147760366Z INFO: Waiting for node performance related collection to complete ... 2026-07-13T20:34:37.831225964Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-07-13T20:34:37.831270519Z INFO: on-disk MachineConfig config collection complete. 2026-07-13T20:34:37.937257842Z INFO: Found 1 replicas - prometheus-k8s-0 2026-07-13T20:34:37.951003835Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-m9x8r 2026-07-13T20:34:37.951003835Z insights-runtime-extractor-948n4 2026-07-13T20:34:37.951003835Z insights-runtime-extractor-mn5j9 2026-07-13T20:34:37.951003835Z insights-runtime-extractor-q8vn2 2026-07-13T20:34:38.163024093Z INFO: "metallb-operator" not detected. Skipping. 2026-07-13T20:34:38.224249705Z No resources found 2026-07-13T20:34:38.269535100Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-07-13T20:34:38.488834761Z No resources found in openshift-etcd namespace. 2026-07-13T20:34:38.591297018Z error: the server doesn't have a resource type "performanceprofile" 2026-07-13T20:34:38.602667826Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-07-13T20:34:38.650863843Z Wrote inspect data to must-gather. 2026-07-13T20:34:38.680234022Z INFO: Worker host service log collection to complete. 2026-07-13T20:34:38.680477602Z INFO: "sriov-network-operator" not detected. Skipping. 2026-07-13T20:34:38.782044788Z Wrote inspect data to must-gather. 2026-07-13T20:34:39.018871328Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-07-13T20:34:39.018871328Z See 'oc rsync -h' for help and examples 2026-07-13T20:34:39.068990678Z INFO: Waiting for HAProxy config collection to complete ... 2026-07-13T20:34:39.194579049Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-13T20:34:39.615420046Z ERROR: No running kube-apiserver pods found 2026-07-13T20:34:39.648883848Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-07-13T20:34:39.690002732Z Gathering data for ns/openshift-monitoring... 2026-07-13T20:34:39.693773868Z Wrote inspect data to must-gather. 2026-07-13T20:34:39.702432127Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-07-13T20:34:39.720113466Z No resources found 2026-07-13T20:34:39.720280812Z Wrote inspect data to must-gather. 2026-07-13T20:34:39.721608419Z Wrote inspect data to must-gather. 2026-07-13T20:34:39.830862762Z Wrote inspect data to must-gather. 2026-07-13T20:34:39.843732712Z Wrote inspect data to must-gather. 2026-07-13T20:34:39.858776725Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-13T20:34:39.874957486Z INFO: 'previous-logs' folder not found on ip-10-0-130-180.ec2.internal, skipping... 2026-07-13T20:34:39.887590113Z tar: Removing leading `/' from member names 2026-07-13T20:34:39.893699146Z INFO: HAProxy config collection complete. 2026-07-13T20:34:40.108798831Z error: the server doesn't have a resource type "machineconfigs" 2026-07-13T20:34:40.162207396Z Wrote inspect data to must-gather. 2026-07-13T20:34:40.241180338Z INFO: 'previous-logs' folder not found on ip-10-0-133-78.ec2.internal, skipping... 2026-07-13T20:34:40.326709081Z error: the server doesn't have a resource type "machineconfigpools" 2026-07-13T20:34:40.341368144Z INFO: Getting rules from prometheus-k8s-0 2026-07-13T20:34:40.505472315Z Wrote inspect data to must-gather. 2026-07-13T20:34:40.563358472Z INFO: 'previous-logs' folder not found on ip-10-0-140-94.ec2.internal, skipping... 2026-07-13T20:34:40.563535699Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-07-13T20:34:40.563618089Z INFO: Machine Config Daemon termination log collection complete. 2026-07-13T20:34:40.647091812Z Wrote inspect data to must-gather. 2026-07-13T20:34:40.811464530Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-07-13T20:34:40.844857069Z Wrote inspect data to must-gather. 2026-07-13T20:34:40.858594130Z error: the server doesn't have a resource type "kubeletconfigs" 2026-07-13T20:34:40.873078536Z INFO: Getting status/config from prometheus-k8s-0 2026-07-13T20:34:41.111898539Z Wrote inspect data to must-gather. 2026-07-13T20:34:41.172596784Z INFO: INTERCONNECT MODE 2026-07-13T20:34:41.172667714Z INFO: Gathering ovn-kubernetes DBs 2026-07-13T20:34:41.365733708Z INFO: Gathering OVN_Northbound from ovnkube-node-48dtn... 2026-07-13T20:34:41.366010142Z INFO: Gathering OVN_Northbound from ovnkube-node-gf9b8... 2026-07-13T20:34:41.366269709Z INFO: Gathering OVN_Northbound from ovnkube-node-r79z2... 2026-07-13T20:34:41.366570149Z INFO: Gathering OVN_Southbound from ovnkube-node-48dtn... 2026-07-13T20:34:41.366871969Z INFO: Gathering OVN_Southbound from ovnkube-node-gf9b8... 2026-07-13T20:34:41.367343989Z INFO: Gathering OVN_Southbound from ovnkube-node-r79z2... 2026-07-13T20:34:41.368646283Z INFO: Getting status/flags from prometheus-k8s-0 2026-07-13T20:34:41.372180438Z Wrote inspect data to must-gather. 2026-07-13T20:34:41.718786671Z tar: Removing leading `/' from member names 2026-07-13T20:34:41.785842925Z tar: Removing leading `/' from member names 2026-07-13T20:34:41.788143330Z tar: Removing leading `/' from member names 2026-07-13T20:34:41.789404632Z tar: Removing leading `/' from member names 2026-07-13T20:34:41.806612840Z tar: Removing leading `/' from member names 2026-07-13T20:34:41.814794992Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-07-13T20:34:41.826250246Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-07-13T20:34:41.844056478Z tar: Removing leading `/' from member names 2026-07-13T20:34:41.852931885Z Wrote inspect data to must-gather. 2026-07-13T20:34:41.949138071Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-07-13T20:34:42.110588874Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-07-13T20:34:42.142473008Z Wrote inspect data to must-gather. 2026-07-13T20:34:42.219844508Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:42.414296705Z W0713 20:34:42.414252 1435 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' 2026-07-13T20:34:42.421188930Z Wrote inspect data to must-gather. 2026-07-13T20:34:42.422430231Z INFO: Waiting for network log collection to complete ... 2026-07-13T20:34:42.422480537Z INFO: Waiting for ovnk database copies to complete ... 2026-07-13T20:34:42.422999486Z INFO: Copying ovnk databases complete. 2026-07-13T20:34:42.424871902Z 3.1M must-gather/network_logs/ovnk_database_store 2026-07-13T20:34:42.427569605Z ovnk_database_store/ 2026-07-13T20:34:42.427569605Z ovnk_database_store/ovnkube-node-gf9b8_nbdb 2026-07-13T20:34:42.428625060Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-07-13T20:34:42.438012326Z ovnk_database_store/ovnkube-node-r79z2_sbdb 2026-07-13T20:34:42.461030710Z ovnk_database_store/ovnkube-node-48dtn_nbdb 2026-07-13T20:34:42.468812436Z ovnk_database_store/ovnkube-node-r79z2_nbdb 2026-07-13T20:34:42.479590970Z ovnk_database_store/ovnkube-node-gf9b8_sbdb 2026-07-13T20:34:42.497806792Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:42.497840019Z ovnk_database_store/ovnkube-node-48dtn_sbdb 2026-07-13T20:34:42.504128798Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-07-13T20:34:42.518820194Z INFO: Network log collection complete. 2026-07-13T20:34:42.540868190Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-07-13T20:34:42.770766220Z Gathering data for ns/openshift-network-console... 2026-07-13T20:34:42.774873572Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:42.783882674Z Wrote inspect data to must-gather. 2026-07-13T20:34:42.919994723Z INFO: Getting status from alertmanager-main-0 2026-07-13T20:34:43.017034306Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:43.022644523Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-07-13T20:34:43.134961783Z Wrote inspect data to must-gather. 2026-07-13T20:34:43.253664866Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:43.307505341Z Gathering data for ns/openshift-console-operator... 2026-07-13T20:34:43.459255050Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-07-13T20:34:43.461475455Z INFO: Fallback to identify the container image from release info 2026-07-13T20:34:43.483254152Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:43.486481757Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-07-13T20:34:43.743868925Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:43.749838548Z Gathering data for ns/openshift-console... 2026-07-13T20:34:43.955037110Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:43.957332022Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io 2026-07-13T20:34:44.137112637Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:44.220160118Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d 2026-07-13T20:34:44.291972490Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-13T20:34:44.323309556Z Gathering data for ns/openshift-cluster-storage-operator... 2026-07-13T20:34:44.379340599Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:44.382867067Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io 2026-07-13T20:34:44.386265133Z daemonset.apps/perf-node-gather-daemonset created 2026-07-13T20:34:44.550193479Z Waiting for performance profile collector pods to become ready: 1 2026-07-13T20:34:44.592345910Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:44.817276922Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:44.823076759Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io 2026-07-13T20:34:44.965888559Z Gathering data for ns/openshift-dns-operator... 2026-07-13T20:34:45.064542105Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:45.273079270Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:45.276324246Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-07-13T20:34:45.317759598Z Gathering data for ns/openshift-dns... 2026-07-13T20:34:45.450168471Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:45.676200428Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:45.678967334Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-07-13T20:34:45.721249546Z Waiting for performance profile collector pods to become ready: 2 2026-07-13T20:34:45.860848985Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:46.018905536Z Gathering data for ns/openshift-image-registry... 2026-07-13T20:34:46.084248728Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:46.087243150Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-07-13T20:34:46.269407695Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:46.485061274Z Wrote inspect data to /must-gather/istio. 2026-07-13T20:34:46.548221500Z Gathering data for ns/openshift-ingress-operator... 2026-07-13T20:34:46.878123120Z Waiting for performance profile collector pods to become ready: 3 2026-07-13T20:34:46.945335959Z Gathering data for ns/openshift-ingress... 2026-07-13T20:34:47.273179774Z Done executing Istio gather script 2026-07-13T20:34:47.388296975Z Gathering data for ns/openshift-ingress-canary... 2026-07-13T20:34:47.816792485Z Gathering data for ns/openshift-insights... 2026-07-13T20:34:48.001785333Z Waiting for performance profile collector pods to become ready: 4 2026-07-13T20:34:49.135825235Z Waiting for performance profile collector pods to become ready: 5 2026-07-13T20:34:49.297410260Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-13T20:34:50.144856050Z Gathering data for ns/kserve... 2026-07-13T20:34:50.256250757Z Waiting for performance profile collector pods to become ready: 6 2026-07-13T20:34:51.391516279Z Waiting for performance profile collector pods to become ready: 7 2026-07-13T20:34:51.843875859Z Gathering data for ns/openshift-config... 2026-07-13T20:34:52.163372974Z Gathering data for ns/openshift-config-managed... 2026-07-13T20:34:52.524921645Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-07-13T20:34:52.623748876Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-07-13T20:34:52.825464928Z Collecting performance related data for node ip-10-0-140-94.ec2.internal 2026-07-13T20:34:52.836657737Z Collecting performance related data for node ip-10-0-133-78.ec2.internal 2026-07-13T20:34:52.847248760Z Collecting performance related data for node ip-10-0-130-180.ec2.internal 2026-07-13T20:34:52.974634044Z Gathering data for ns/openshift-kube-apiserver... 2026-07-13T20:34:53.310741789Z Gathering data for ns/openshift-kube-controller-manager... 2026-07-13T20:34:53.690285291Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-07-13T20:34:54.101853231Z Gathering data for ns/openshift-kube-scheduler... 2026-07-13T20:34:54.303276444Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-13T20:34:54.496631233Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-07-13T20:34:54.805412680Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-07-13T20:34:55.147266960Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-07-13T20:34:55.585977149Z Gathering data for ns/openshift-user-workload-monitoring... 2026-07-13T20:34:56.162936401Z Gathering data for ns/openshift-multus... 2026-07-13T20:34:57.760065953Z Gathering data for ns/openshift-ovn-kubernetes... 2026-07-13T20:34:59.083660784Z Gathering data for ns/openshift-host-network... 2026-07-13T20:34:59.311952278Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-13T20:34:59.458949836Z Gathering data for ns/openshift-network-diagnostics... 2026-07-13T20:35:00.043709901Z Gathering data for ns/openshift-network-node-identity... 2026-07-13T20:35:00.522922953Z Gathering data for ns/openshift-network-operator... 2026-07-13T20:35:00.954430647Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-07-13T20:35:01.258078546Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-07-13T20:35:01.330540758Z Collecting kubelet logs for node ip-10-0-140-94.ec2.internal 2026-07-13T20:35:01.332522908Z Collecting kubelet logs for node ip-10-0-133-78.ec2.internal 2026-07-13T20:35:01.334360020Z Collecting kubelet logs for node ip-10-0-130-180.ec2.internal 2026-07-13T20:35:01.715632884Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-d42b5 namespace 2026-07-13T20:35:01.721362240Z INFO: Node performance data collection complete. 2026-07-13T20:35:01.870984724Z Gathering data for ns/openshift-apiserver-operator... 2026-07-13T20:35:02.236976017Z Gathering data for ns/openshift-apiserver... 2026-07-13T20:35:02.664513777Z Gathering data for ns/openshift-controller-manager-operator... 2026-07-13T20:35:02.958149726Z Gathering data for ns/openshift-controller-manager... 2026-07-13T20:35:03.252855925Z Gathering data for ns/openshift-cluster-samples-operator... 2026-07-13T20:35:03.927365532Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-07-13T20:35:04.288911386Z Gathering data for ns/openshift-service-ca-operator... 2026-07-13T20:35:04.316903608Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70 2026-07-13T20:35:04.624879836Z Gathering data for ns/openshift-service-ca... 2026-07-13T20:35:05.013885611Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-07-13T20:35:05.956267386Z Wrote inspect data to must-gather. 2026-07-13T20:35:05.956320760Z error: inspection completed with the errors occurred while gathering data: 2026-07-13T20:35:05.956320760Z [skipping gathering secrets/support due to error: secrets "support" not found, skipping gathering customresourcedefinitions.apiextensions.k8s.io due to error: skipping gathering namespaces/kserve due to error: one or more errors occurred while gathering pod-specific data for namespace: kserve 2026-07-13T20:35:05.956320760Z 2026-07-13T20:35:05.956320760Z one or more errors occurred while gathering container data for pod s3-tls-init-serving-6tcpq: 2026-07-13T20:35:05.956320760Z 2026-07-13T20:35:05.956320760Z [container "s3-tls-init-serving" in pod "s3-tls-init-serving-6tcpq" is waiting to start: PodInitializing, previous terminated container "s3-tls-init-serving" in pod "s3-tls-init-serving-6tcpq" not found, the server could not find the requested resource]] 2026-07-13T20:35:06.094454037Z error: the server doesn't have a resource type "clusters" 2026-07-13T20:35:06.236969936Z Caches written to disk