2026-08-04T18:27:09.544756048Z [disk usage checker] Started 2026-08-04T18:27:09.547900011Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 2026-08-04T18:27:09.739528115Z Gathering data for ns/openshift-cluster-version... 2026-08-04T18:27:09.891683175Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-08-04T18:27:09.986991547Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-08-04T18:27:10.080274048Z Gathering data for ns/default... 2026-08-04T18:27:10.457487846Z Gathering data for ns/openshift... 2026-08-04T18:27:10.878468904Z Gathering data for ns/kube-system... 2026-08-04T18:27:11.564653342Z Gathering data for ns/openshift-etcd... 2026-08-04T18:27:11.886163232Z Wrote inspect data to must-gather. 2026-08-04T18:27:11.886196992Z error: inspection completed with the errors occurred while gathering data: 2026-08-04T18:27:11.886196992Z namespaces "assisted-installer" not found 2026-08-04T18:27:12.200841795Z Waiting on subprocesses to finish execution. 2026-08-04T18:27:12.211128035Z INFO: Gathering HAProxy config files 2026-08-04T18:27:12.212697720Z INFO: Gathering machine config daemon's old logs from all nodes 2026-08-04T18:27:12.222156087Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-08-04T18:27:12.224822638Z Executing Istio gather script 2026-08-04T18:27:12.235341448Z INFO: Collecting host service logs for crio 2026-08-04T18:27:12.235705054Z INFO: Collecting host service logs for kubelet 2026-08-04T18:27:12.236056656Z INFO: Collecting host service logs for rpm-ostreed 2026-08-04T18:27:12.236420589Z INFO: Collecting host service logs for ostree-finalize-staged 2026-08-04T18:27:12.236739837Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-08-04T18:27:12.237056823Z INFO: Collecting host service logs for machine-config-daemon-host 2026-08-04T18:27:12.237465992Z INFO: Collecting host service logs for NetworkManager 2026-08-04T18:27:12.237627035Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-08-04T18:27:12.237827278Z INFO: Collecting host service logs for openvswitch 2026-08-04T18:27:12.238134840Z INFO: Collecting host service logs for ovs-configuration 2026-08-04T18:27:12.238495017Z INFO: Collecting host service logs for ovsdb-server 2026-08-04T18:27:12.238836465Z INFO: Collecting host service logs for ovs-vswitchd 2026-08-04T18:27:12.239207890Z INFO: Waiting for worker host service log collection to complete ... 2026-08-04T18:27:12.303806529Z INFO: Waiting for node performance related collection to complete ... 2026-08-04T18:27:12.741422655Z No resources found 2026-08-04T18:27:12.834285024Z INFO: Collecting Insights Archives from insights-operator-77696685b-cvxwr 2026-08-04T18:27:12.834342094Z insights-runtime-extractor-gwkr7 2026-08-04T18:27:12.834363969Z insights-runtime-extractor-j84dm 2026-08-04T18:27:12.834383037Z insights-runtime-extractor-w9k9t 2026-08-04T18:27:12.850170847Z Wrote inspect data to must-gather. 2026-08-04T18:27:12.921822187Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-08-04T18:27:12.922281752Z INFO: "sriov-network-operator" not detected. Skipping. 2026-08-04T18:27:13.058982233Z No resources found in openshift-etcd namespace. 2026-08-04T18:27:13.282778964Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-08-04T18:27:13.282778964Z See 'oc rsync -h' for help and examples 2026-08-04T18:27:13.283470470Z INFO: "metallb-operator" not detected. Skipping. 2026-08-04T18:27:13.399052011Z ERROR: No running kube-apiserver pods found 2026-08-04T18:27:13.469748253Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-08-04T18:27:13.518565376Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-08-04T18:27:13.518600178Z INFO: on-disk MachineConfig config collection complete. 2026-08-04T18:27:13.594499170Z INFO: Waiting for HAProxy config collection to complete ... 2026-08-04T18:27:13.600444585Z INFO: Found 1 replicas - prometheus-k8s-0 2026-08-04T18:27:13.603606894Z INFO: Worker host service log collection to complete. 2026-08-04T18:27:13.862391699Z INFO: 'previous-logs' folder not found on ip-10-0-128-51.ec2.internal, skipping... 2026-08-04T18:27:13.983029500Z error: the server doesn't have a resource type "performanceprofile" 2026-08-04T18:27:14.195685258Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-08-04T18:27:14.256591797Z tar: Removing leading `/' from member names 2026-08-04T18:27:14.263036024Z INFO: HAProxy config collection complete. 2026-08-04T18:27:14.287984877Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-08-04T18:27:14.360403778Z Wrote inspect data to must-gather. 2026-08-04T18:27:14.404982516Z Wrote inspect data to must-gather. 2026-08-04T18:27:14.567564486Z Wrote inspect data to must-gather. 2026-08-04T18:27:14.576219209Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 2026-08-04T18:27:14.580136920Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-08-04T18:27:14.599373131Z Wrote inspect data to must-gather. 2026-08-04T18:27:14.629049879Z No resources found 2026-08-04T18:27:14.744691289Z Wrote inspect data to must-gather. 2026-08-04T18:27:14.754536386Z INFO: 'previous-logs' folder not found on ip-10-0-139-237.ec2.internal, skipping... 2026-08-04T18:27:14.796901603Z Wrote inspect data to must-gather. 2026-08-04T18:27:14.948926835Z Wrote inspect data to must-gather. 2026-08-04T18:27:14.973330069Z Gathering data for ns/openshift-monitoring... 2026-08-04T18:27:14.992882458Z error: the server doesn't have a resource type "machineconfigs" 2026-08-04T18:27:15.104719748Z INFO: 'previous-logs' folder not found on ip-10-0-140-15.ec2.internal, skipping... 2026-08-04T18:27:15.104836623Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-08-04T18:27:15.104896931Z INFO: Machine Config Daemon termination log collection complete. 2026-08-04T18:27:15.126965163Z INFO: Getting rules from prometheus-k8s-0 2026-08-04T18:27:15.212664058Z error: the server doesn't have a resource type "machineconfigpools" 2026-08-04T18:27:15.263988523Z Wrote inspect data to must-gather. 2026-08-04T18:27:15.473205375Z Wrote inspect data to must-gather. 2026-08-04T18:27:15.559284809Z Wrote inspect data to must-gather. 2026-08-04T18:27:15.581016305Z INFO: Getting status/config from prometheus-k8s-0 2026-08-04T18:27:15.654477091Z error: the server doesn't have a resource type "kubeletconfigs" 2026-08-04T18:27:15.757430084Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-08-04T18:27:15.871500748Z Wrote inspect data to must-gather. 2026-08-04T18:27:15.882504532Z INFO: INTERCONNECT MODE 2026-08-04T18:27:15.882504532Z INFO: Gathering ovn-kubernetes DBs 2026-08-04T18:27:16.025176730Z INFO: Getting status/flags from prometheus-k8s-0 2026-08-04T18:27:16.051791741Z INFO: Gathering OVN_Northbound from ovnkube-node-dwhtp... 2026-08-04T18:27:16.052065620Z INFO: Gathering OVN_Northbound from ovnkube-node-njg2j... 2026-08-04T18:27:16.052305280Z INFO: Gathering OVN_Northbound from ovnkube-node-tkvl9... 2026-08-04T18:27:16.052599517Z INFO: Gathering OVN_Southbound from ovnkube-node-dwhtp... 2026-08-04T18:27:16.052888374Z INFO: Gathering OVN_Southbound from ovnkube-node-njg2j... 2026-08-04T18:27:16.053130725Z INFO: Gathering OVN_Southbound from ovnkube-node-tkvl9... 2026-08-04T18:27:16.099433458Z Wrote inspect data to must-gather. 2026-08-04T18:27:16.405440380Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-08-04T18:27:16.433098136Z tar: Removing leading `/' from member names 2026-08-04T18:27:16.434484565Z tar: Removing leading `/' from member names 2026-08-04T18:27:16.440458295Z tar: Removing leading `/' from member names 2026-08-04T18:27:16.471298774Z tar: Removing leading `/' from member names 2026-08-04T18:27:16.496246860Z tar: Removing leading `/' from member names 2026-08-04T18:27:16.506079099Z tar: Removing leading `/' from member names 2026-08-04T18:27:16.534452634Z Wrote inspect data to must-gather. 2026-08-04T18:27:16.617441123Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-08-04T18:27:16.699619024Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-08-04T18:27:16.804525196Z Wrote inspect data to must-gather. 2026-08-04T18:27:16.816088383Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-08-04T18:27:16.838886106Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:16.980014890Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-08-04T18:27:16.983826023Z INFO: Waiting for network log collection to complete ... 2026-08-04T18:27:16.984457395Z INFO: Waiting for ovnk database copies to complete ... 2026-08-04T18:27:16.984949631Z INFO: Copying ovnk databases complete. 2026-08-04T18:27:16.988240691Z 3.5M must-gather/network_logs/ovnk_database_store 2026-08-04T18:27:16.990531341Z ovnk_database_store/ 2026-08-04T18:27:16.990531341Z ovnk_database_store/ovnkube-node-tkvl9_sbdb 2026-08-04T18:27:17.014552328Z ovnk_database_store/ovnkube-node-njg2j_sbdb 2026-08-04T18:27:17.035971688Z ovnk_database_store/ovnkube-node-njg2j_nbdb 2026-08-04T18:27:17.039624609Z Wrote inspect data to must-gather. 2026-08-04T18:27:17.045597167Z ovnk_database_store/ovnkube-node-tkvl9_nbdb 2026-08-04T18:27:17.055117011Z ovnk_database_store/ovnkube-node-dwhtp_nbdb 2026-08-04T18:27:17.063740313Z ovnk_database_store/ovnkube-node-dwhtp_sbdb 2026-08-04T18:27:17.087015650Z INFO: Network log collection complete. 2026-08-04T18:27:17.097342027Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:17.099506223Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-08-04T18:27:17.273289090Z Wrote inspect data to must-gather. 2026-08-04T18:27:17.330375407Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:17.433396287Z INFO: Getting status from alertmanager-main-0 2026-08-04T18:27:17.477171297Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-08-04T18:27:17.589493858Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:17.593114350Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-08-04T18:27:17.609300472Z Wrote inspect data to must-gather. 2026-08-04T18:27:17.689855173Z Gathering data for ns/openshift-network-console... 2026-08-04T18:27:17.808909760Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:17.896956068Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-08-04T18:27:17.900210019Z INFO: Fallback to identify the container image from release info 2026-08-04T18:27:18.013784800Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:18.016623943Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-08-04T18:27:18.214965704Z Gathering data for ns/openshift-console-operator... 2026-08-04T18:27:18.243957810Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:18.428353645Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:18.430489792Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-08-04T18:27:18.457851437Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2eb1033f55d7129b37ecfc3e4e9a306e42e5ab3dd7d2fe0ff08c8b45ea2b675 2026-08-04T18:27:18.592346601Z daemonset.apps/perf-node-gather-daemonset created 2026-08-04T18:27:18.619408992Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:18.676159897Z Gathering data for ns/openshift-console... 2026-08-04T18:27:18.728847894Z Waiting for performance profile collector pods to become ready: 1 2026-08-04T18:27:18.828097941Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:18.831460173Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-08-04T18:27:18.989832237Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:19.220498685Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:19.223518436Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-08-04T18:27:19.275650787Z Gathering data for ns/openshift-cluster-storage-operator... 2026-08-04T18:27:19.391896082Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:19.571716254Z Wrote inspect data to /must-gather/istio. 2026-08-04T18:27:19.581968322Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 2026-08-04T18:27:19.818023870Z Gathering data for ns/openshift-dns-operator... 2026-08-04T18:27:19.872081984Z Waiting for performance profile collector pods to become ready: 2 2026-08-04T18:27:20.162009528Z Gathering data for ns/openshift-dns... 2026-08-04T18:27:20.248367799Z Done executing Istio gather script 2026-08-04T18:27:20.753985244Z Gathering data for ns/openshift-image-registry... 2026-08-04T18:27:20.991743055Z Waiting for performance profile collector pods to become ready: 3 2026-08-04T18:27:21.282811327Z Gathering data for ns/openshift-ingress-operator... 2026-08-04T18:27:21.609095176Z Gathering data for ns/openshift-ingress... 2026-08-04T18:27:21.944698932Z Gathering data for ns/openshift-ingress-canary... 2026-08-04T18:27:22.110978041Z Waiting for performance profile collector pods to become ready: 4 2026-08-04T18:27:22.380113941Z Gathering data for ns/openshift-insights... 2026-08-04T18:27:23.222933599Z Waiting for performance profile collector pods to become ready: 5 2026-08-04T18:27:24.337073127Z Waiting for performance profile collector pods to become ready: 6 2026-08-04T18:27:24.586979750Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 2026-08-04T18:27:24.701984068Z Gathering data for ns/kserve... 2026-08-04T18:27:25.463241320Z Waiting for performance profile collector pods to become ready: 7 2026-08-04T18:27:26.323071369Z Gathering data for ns/openshift-config... 2026-08-04T18:27:26.587277978Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-08-04T18:27:26.671676274Z Gathering data for ns/openshift-config-managed... 2026-08-04T18:27:26.836491117Z Collecting performance related data for node ip-10-0-139-237.ec2.internal 2026-08-04T18:27:26.849291125Z Collecting performance related data for node ip-10-0-140-15.ec2.internal 2026-08-04T18:27:26.854846922Z Collecting performance related data for node ip-10-0-128-51.ec2.internal 2026-08-04T18:27:27.163977550Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-08-04T18:27:27.513782083Z Gathering data for ns/openshift-kube-apiserver... 2026-08-04T18:27:27.853843285Z Gathering data for ns/openshift-kube-controller-manager... 2026-08-04T18:27:28.306748302Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-08-04T18:27:28.710807705Z Gathering data for ns/openshift-kube-scheduler... 2026-08-04T18:27:28.977918113Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-08-04T18:27:29.288183750Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-08-04T18:27:29.592013804Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 2026-08-04T18:27:29.698934536Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-08-04T18:27:30.057907860Z Gathering data for ns/openshift-user-workload-monitoring... 2026-08-04T18:27:30.590950234Z Gathering data for ns/openshift-multus... 2026-08-04T18:27:32.043951524Z Gathering data for ns/openshift-ovn-kubernetes... 2026-08-04T18:27:33.309329251Z Gathering data for ns/openshift-host-network... 2026-08-04T18:27:33.685901750Z Gathering data for ns/openshift-network-diagnostics... 2026-08-04T18:27:34.234491855Z Gathering data for ns/openshift-network-node-identity... 2026-08-04T18:27:34.598873193Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 2026-08-04T18:27:34.647596311Z Gathering data for ns/openshift-network-operator... 2026-08-04T18:27:35.034642566Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-08-04T18:27:35.067779661Z Collecting kubelet logs for node ip-10-0-139-237.ec2.internal 2026-08-04T18:27:35.070407432Z Collecting kubelet logs for node ip-10-0-140-15.ec2.internal 2026-08-04T18:27:35.072636645Z Collecting kubelet logs for node ip-10-0-128-51.ec2.internal 2026-08-04T18:27:35.359558396Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-08-04T18:27:35.434205959Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-flh2v namespace 2026-08-04T18:27:35.439013193Z INFO: Node performance data collection complete. 2026-08-04T18:27:35.860065312Z Gathering data for ns/openshift-apiserver-operator... 2026-08-04T18:27:36.201365882Z Gathering data for ns/openshift-apiserver... 2026-08-04T18:27:36.642608804Z Gathering data for ns/openshift-controller-manager-operator... 2026-08-04T18:27:36.904983624Z Gathering data for ns/openshift-controller-manager... 2026-08-04T18:27:37.171391199Z Gathering data for ns/openshift-cluster-samples-operator... 2026-08-04T18:27:37.777099004Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-08-04T18:27:38.121542467Z Gathering data for ns/openshift-service-ca-operator... 2026-08-04T18:27:38.453105199Z Gathering data for ns/openshift-service-ca... 2026-08-04T18:27:38.825016336Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-08-04T18:27:39.603475003Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 2026-08-04T18:27:39.722187105Z Wrote inspect data to must-gather. 2026-08-04T18:27:39.722225836Z error: inspection completed with the errors occurred while gathering data: 2026-08-04T18:27:39.722225836Z skipping gathering secrets/support due to error: secrets "support" not found 2026-08-04T18:27:39.839913058Z error: the server doesn't have a resource type "clusters" 2026-08-04T18:27:39.993427176Z Caches written to disk