2026-07-28T16:37:48.653569773Z [disk usage checker] Started
2026-07-28T16:37:48.656921953Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:37:48.854478786Z Gathering data for ns/openshift-cluster-version...
2026-07-28T16:37:49.019022874Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T16:37:49.132784453Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:37:49.238160285Z Gathering data for ns/default...
2026-07-28T16:37:49.592109822Z Gathering data for ns/openshift...
2026-07-28T16:37:50.099592651Z Gathering data for ns/kube-system...
2026-07-28T16:37:51.012227683Z Gathering data for ns/openshift-etcd...
2026-07-28T16:37:51.367652510Z Wrote inspect data to must-gather.
2026-07-28T16:37:51.367680413Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T16:37:51.367680413Z     namespaces "assisted-installer" not found
2026-07-28T16:37:51.499586340Z Waiting on subprocesses to finish execution.
2026-07-28T16:37:51.508257073Z INFO: Gathering HAProxy config files
2026-07-28T16:37:51.514053905Z INFO: Collecting host service logs for crio
2026-07-28T16:37:51.514314415Z INFO: Collecting host service logs for kubelet
2026-07-28T16:37:51.514586660Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T16:37:51.514607687Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T16:37:51.514912693Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T16:37:51.515213685Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T16:37:51.515503999Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T16:37:51.515807916Z INFO: Collecting host service logs for NetworkManager
2026-07-28T16:37:51.516116175Z INFO: Collecting host service logs for openvswitch
2026-07-28T16:37:51.516383324Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T16:37:51.516733017Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T16:37:51.517056247Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T16:37:51.517408066Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T16:37:51.518949193Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T16:37:51.520392145Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T16:37:51.551102488Z Executing Istio gather script
2026-07-28T16:37:51.556154433Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T16:37:51.961099261Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T16:37:52.352541117Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T16:37:52.410863848Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T16:37:52.410949323Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T16:37:52.474709788Z No resources found in openshift-etcd namespace.
2026-07-28T16:37:52.547551544Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T16:37:52.709464407Z Wrote inspect data to must-gather.
2026-07-28T16:37:52.711758589Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T16:37:52.886299804Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T16:37:53.028040888Z ERROR: No running kube-apiserver pods found
2026-07-28T16:37:53.102249221Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-lxlwg
2026-07-28T16:37:53.102319743Z insights-runtime-extractor-8r9q6
2026-07-28T16:37:53.102341405Z insights-runtime-extractor-pfgp6
2026-07-28T16:37:53.102362027Z insights-runtime-extractor-rq7b2
2026-07-28T16:37:53.111196901Z No resources found
2026-07-28T16:37:53.117309477Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T16:37:53.192730677Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T16:37:53.226368086Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T16:37:53.325155248Z Wrote inspect data to must-gather.
2026-07-28T16:37:53.339223587Z INFO: Worker host service log collection to complete.
2026-07-28T16:37:53.567365802Z Wrote inspect data to must-gather.
2026-07-28T16:37:53.749632788Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:37:53.939107989Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:37:53.951237771Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T16:37:53.951237771Z See 'oc rsync -h' for help and examples
2026-07-28T16:37:54.012672853Z Wrote inspect data to must-gather.
2026-07-28T16:37:54.013536641Z INFO: 'previous-logs' folder not found on ip-10-0-131-181.ec2.internal, skipping...
2026-07-28T16:37:54.116381506Z Wrote inspect data to must-gather.
2026-07-28T16:37:54.125461296Z Wrote inspect data to must-gather.
2026-07-28T16:37:54.167053745Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T16:37:54.186521831Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T16:37:54.289369559Z No resources found
2026-07-28T16:37:54.326867098Z tar: Removing leading `/' from member names
2026-07-28T16:37:54.336307904Z INFO: HAProxy config collection complete.
2026-07-28T16:37:54.356931565Z Wrote inspect data to must-gather.
2026-07-28T16:37:54.491081978Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T16:37:54.518116494Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T16:37:54.612252757Z INFO: 'previous-logs' folder not found on ip-10-0-138-10.ec2.internal, skipping...
2026-07-28T16:37:54.713624764Z Wrote inspect data to must-gather.
2026-07-28T16:37:54.758433250Z Gathering data for ns/openshift-monitoring...
2026-07-28T16:37:54.770126310Z Wrote inspect data to must-gather.
2026-07-28T16:37:54.922395612Z INFO: 'previous-logs' folder not found on ip-10-0-142-59.ec2.internal, skipping...
2026-07-28T16:37:54.922506393Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T16:37:54.922556352Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T16:37:54.973884666Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T16:37:55.044669697Z Wrote inspect data to must-gather.
2026-07-28T16:37:55.048759106Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T16:37:55.236919863Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T16:37:55.242814139Z Wrote inspect data to must-gather.
2026-07-28T16:37:55.325012357Z Wrote inspect data to must-gather.
2026-07-28T16:37:55.472153784Z Wrote inspect data to must-gather.
2026-07-28T16:37:55.522668626Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T16:37:55.638444644Z INFO: INTERCONNECT MODE
2026-07-28T16:37:55.638591019Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T16:37:55.732371262Z Wrote inspect data to must-gather.
2026-07-28T16:37:55.806685234Z INFO: Gathering OVN_Northbound from ovnkube-node-4xnwq...
2026-07-28T16:37:55.806927620Z INFO: Gathering OVN_Northbound from ovnkube-node-khj85...
2026-07-28T16:37:55.807176989Z INFO: Gathering OVN_Northbound from ovnkube-node-rc9lx...
2026-07-28T16:37:55.807238777Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T16:37:55.807434731Z INFO: Gathering OVN_Southbound from ovnkube-node-4xnwq...
2026-07-28T16:37:55.807731227Z INFO: Gathering OVN_Southbound from ovnkube-node-khj85...
2026-07-28T16:37:55.808050645Z INFO: Gathering OVN_Southbound from ovnkube-node-rc9lx...
2026-07-28T16:37:56.061804621Z Wrote inspect data to must-gather.
2026-07-28T16:37:56.147728601Z tar: Removing leading `/' from member names
2026-07-28T16:37:56.157221010Z tar: Removing leading `/' from member names
2026-07-28T16:37:56.160405360Z tar: Removing leading `/' from member names
2026-07-28T16:37:56.162068561Z tar: Removing leading `/' from member names
2026-07-28T16:37:56.207017015Z tar: Removing leading `/' from member names
2026-07-28T16:37:56.207418457Z tar: Removing leading `/' from member names
2026-07-28T16:37:56.294231759Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T16:37:56.317897076Z Wrote inspect data to must-gather.
2026-07-28T16:37:56.554806069Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T16:37:56.563455200Z Wrote inspect data to must-gather.
2026-07-28T16:37:56.598831361Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T16:37:56.671918458Z INFO: Waiting for network log collection to complete ...
2026-07-28T16:37:56.671950481Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T16:37:56.672328773Z INFO: Copying ovnk databases complete.
2026-07-28T16:37:56.674515813Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-28T16:37:56.677210193Z ovnk_database_store/
2026-07-28T16:37:56.677257564Z ovnk_database_store/ovnkube-node-rc9lx_sbdb
2026-07-28T16:37:56.702626649Z ovnk_database_store/ovnkube-node-4xnwq_nbdb
2026-07-28T16:37:56.711653435Z ovnk_database_store/ovnkube-node-4xnwq_sbdb
2026-07-28T16:37:56.732203646Z ovnk_database_store/ovnkube-node-rc9lx_nbdb
2026-07-28T16:37:56.745015510Z ovnk_database_store/ovnkube-node-khj85_nbdb
2026-07-28T16:37:56.752544695Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:56.754438090Z ovnk_database_store/ovnkube-node-khj85_sbdb
2026-07-28T16:37:56.777088188Z INFO: Network log collection complete.
2026-07-28T16:37:56.797868825Z Wrote inspect data to must-gather.
2026-07-28T16:37:56.994288919Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:56.997144126Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T16:37:57.096741953Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T16:37:57.098680838Z INFO: Fallback to identify the container image from release info
2026-07-28T16:37:57.124642586Z INFO: Getting status from alertmanager-main-0
2026-07-28T16:37:57.146953840Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T16:37:57.213143129Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:57.430295123Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:57.432959870Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T16:37:57.636563069Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:57.675316465Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T16:37:57.777994769Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:37:57.816347129Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T16:37:57.860250010Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:57.863643278Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T16:37:57.971672011Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T16:37:57.990435997Z Gathering data for ns/openshift-network-console...
2026-07-28T16:37:58.108096323Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:58.317834516Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:58.321089535Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T16:37:58.509334066Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:58.606805517Z Gathering data for ns/openshift-console-operator...
2026-07-28T16:37:58.714436277Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:58.718339600Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T16:37:58.881320253Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:58.976300317Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:37:59.091324210Z Gathering data for ns/openshift-console...
2026-07-28T16:37:59.095907918Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:59.098704779Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T16:37:59.118749661Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T16:37:59.287921255Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:59.475375974Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:37:59.816793547Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T16:38:00.206878251Z Done executing Istio gather script
2026-07-28T16:38:00.259490408Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T16:38:00.434582690Z Gathering data for ns/openshift-dns-operator...
2026-07-28T16:38:00.775788356Z Gathering data for ns/openshift-dns...
2026-07-28T16:38:01.376602810Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T16:38:01.601840046Z Gathering data for ns/openshift-image-registry...
2026-07-28T16:38:02.176414064Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T16:38:02.509138590Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T16:38:02.584077272Z Gathering data for ns/openshift-ingress...
2026-07-28T16:38:02.974841221Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T16:38:03.431749760Z Gathering data for ns/openshift-insights...
2026-07-28T16:38:03.622073144Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T16:38:03.981130995Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:38:04.747385364Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T16:38:05.878797137Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T16:38:06.021403213Z Gathering data for ns/kserve...
2026-07-28T16:38:06.133041910Z Collecting performance related data for node ip-10-0-138-10.ec2.internal
2026-07-28T16:38:06.140591628Z Collecting performance related data for node ip-10-0-131-181.ec2.internal
2026-07-28T16:38:06.150832463Z Collecting performance related data for node ip-10-0-142-59.ec2.internal
2026-07-28T16:38:08.028279771Z Gathering data for ns/openshift-config...
2026-07-28T16:38:08.375272957Z Gathering data for ns/openshift-config-managed...
2026-07-28T16:38:08.977061447Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T16:38:08.986762100Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:38:09.378277745Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T16:38:09.765085161Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T16:38:10.116471292Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T16:38:10.467183060Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T16:38:10.839739461Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T16:38:11.257229529Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T16:38:11.726012588Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T16:38:12.125728244Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T16:38:12.700148865Z Gathering data for ns/openshift-multus...
2026-07-28T16:38:13.996147935Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:38:14.543026226Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T16:38:14.544614542Z Collecting kubelet logs for node ip-10-0-138-10.ec2.internal
2026-07-28T16:38:14.546324978Z Collecting kubelet logs for node ip-10-0-131-181.ec2.internal
2026-07-28T16:38:14.548418974Z Collecting kubelet logs for node ip-10-0-142-59.ec2.internal
2026-07-28T16:38:14.924976328Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-j7d85 namespace
2026-07-28T16:38:14.936379087Z INFO: Node performance data collection complete.
2026-07-28T16:38:16.188763241Z Gathering data for ns/openshift-host-network...
2026-07-28T16:38:16.564540015Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T16:38:17.306919785Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T16:38:17.840060745Z Gathering data for ns/openshift-network-operator...
2026-07-28T16:38:18.382501427Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T16:38:18.745137240Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T16:38:19.000549169Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:38:19.362280631Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T16:38:19.748008810Z Gathering data for ns/openshift-apiserver...
2026-07-28T16:38:20.191353544Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T16:38:20.497678774Z Gathering data for ns/openshift-controller-manager...
2026-07-28T16:38:20.894387885Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T16:38:21.624872543Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T16:38:21.989077264Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T16:38:22.390927148Z Gathering data for ns/openshift-service-ca...
2026-07-28T16:38:22.840078632Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T16:38:23.898003763Z Wrote inspect data to must-gather.
2026-07-28T16:38:23.898048842Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T16:38:23.898048842Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T16:38:24.008038972Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-07-28T16:38:24.029919999Z error: the server doesn't have a resource type "clusters"
2026-07-28T16:38:24.180674087Z Caches written to disk