2026-08-05T18:04:50.371551536Z [disk usage checker] Started
2026-08-05T18:04:50.374863114Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T18:04:50.560241808Z Gathering data for ns/openshift-cluster-version...
2026-08-05T18:04:50.683009106Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-08-05T18:04:50.773200197Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T18:04:50.851501200Z Gathering data for ns/default...
2026-08-05T18:04:51.128618131Z Gathering data for ns/openshift...
2026-08-05T18:04:51.527633460Z Gathering data for ns/kube-system...
2026-08-05T18:04:52.210097764Z Gathering data for ns/openshift-etcd...
2026-08-05T18:04:52.503088168Z Wrote inspect data to must-gather.
2026-08-05T18:04:52.503120004Z error: inspection completed with the errors occurred while gathering data:
2026-08-05T18:04:52.503120004Z     namespaces "assisted-installer" not found
2026-08-05T18:04:53.053893948Z Waiting on subprocesses to finish execution.
2026-08-05T18:04:53.062756677Z Executing Istio gather script
2026-08-05T18:04:53.069535439Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-08-05T18:04:53.074616593Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-08-05T18:04:53.090001129Z INFO: Gathering HAProxy config files
2026-08-05T18:04:53.097124572Z INFO: Collecting host service logs for crio
2026-08-05T18:04:53.097470610Z INFO: Collecting host service logs for kubelet
2026-08-05T18:04:53.097792352Z INFO: Collecting host service logs for rpm-ostreed
2026-08-05T18:04:53.098110388Z INFO: Collecting host service logs for ostree-finalize-staged
2026-08-05T18:04:53.098449728Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-08-05T18:04:53.098745322Z INFO: Collecting host service logs for machine-config-daemon-host
2026-08-05T18:04:53.099057418Z INFO: Collecting host service logs for NetworkManager
2026-08-05T18:04:53.099367270Z INFO: Collecting host service logs for openvswitch
2026-08-05T18:04:53.099680006Z INFO: Collecting host service logs for ovs-configuration
2026-08-05T18:04:53.099992528Z INFO: Collecting host service logs for ovsdb-server
2026-08-05T18:04:53.100305398Z INFO: Collecting host service logs for ovs-vswitchd
2026-08-05T18:04:53.104750432Z INFO: Waiting for worker host service log collection to complete ...
2026-08-05T18:04:53.114081905Z INFO: Gathering machine config daemon's old logs from all nodes
2026-08-05T18:04:53.208323073Z INFO: Waiting for node performance related collection to complete ...
2026-08-05T18:04:53.519155871Z No resources found
2026-08-05T18:04:53.641146722Z INFO: "sriov-network-operator" not detected. Skipping.
2026-08-05T18:04:53.708452848Z INFO: Collecting Insights Archives from insights-operator-77696685b-svfhd
2026-08-05T18:04:53.708507715Z insights-runtime-extractor-5jjlh
2026-08-05T18:04:53.708528788Z insights-runtime-extractor-hrgfv
2026-08-05T18:04:53.708550100Z insights-runtime-extractor-r2dtl
2026-08-05T18:04:53.808611844Z INFO: Found 1 replicas - prometheus-k8s-0
2026-08-05T18:04:53.831884980Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-08-05T18:04:54.029802045Z No resources found in openshift-etcd namespace.
2026-08-05T18:04:54.040207081Z Wrote inspect data to must-gather.
2026-08-05T18:04:54.365016001Z INFO: "metallb-operator" not detected. Skipping.
2026-08-05T18:04:54.374434293Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-08-05T18:04:54.488814774Z error: the server doesn't have a resource type "performanceprofile"
2026-08-05T18:04:54.491998871Z ERROR: No running kube-apiserver pods found
2026-08-05T18:04:54.499557649Z INFO: Worker host service log collection to complete.
2026-08-05T18:04:54.516507023Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-08-05T18:04:54.516507023Z See 'oc rsync -h' for help and examples
2026-08-05T18:04:54.607235885Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-08-05T18:04:54.607326007Z INFO: on-disk MachineConfig config collection complete.
2026-08-05T18:04:54.691620354Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-08-05T18:04:54.817298644Z INFO: Waiting for HAProxy config collection to complete ...
2026-08-05T18:04:55.019108730Z Wrote inspect data to must-gather.
2026-08-05T18:04:55.124436661Z Wrote inspect data to must-gather.
2026-08-05T18:04:55.142162800Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T18:04:55.337193578Z No resources found
2026-08-05T18:04:55.442200095Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T18:04:55.565669101Z Wrote inspect data to must-gather.
2026-08-05T18:04:55.661604235Z tar: Removing leading `/' from member names
2026-08-05T18:04:55.672514494Z INFO: HAProxy config collection complete.
2026-08-05T18:04:55.708684484Z Wrote inspect data to must-gather.
2026-08-05T18:04:55.723113649Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-08-05T18:04:55.727082467Z Wrote inspect data to must-gather.
2026-08-05T18:04:55.763413629Z Wrote inspect data to must-gather.
2026-08-05T18:04:55.770099668Z INFO: 'previous-logs' folder not found on ip-10-0-135-121.ec2.internal, skipping...
2026-08-05T18:04:55.806816858Z error: the server doesn't have a resource type "machineconfigs"
2026-08-05T18:04:55.824976761Z Gathering data for ns/openshift-monitoring...
2026-08-05T18:04:55.951579375Z INFO: Getting rules from prometheus-k8s-0
2026-08-05T18:04:56.018344670Z error: the server doesn't have a resource type "machineconfigpools"
2026-08-05T18:04:56.058358231Z Wrote inspect data to must-gather.
2026-08-05T18:04:56.082510385Z INFO: 'previous-logs' folder not found on ip-10-0-135-2.ec2.internal, skipping...
2026-08-05T18:04:56.287497456Z Wrote inspect data to must-gather.
2026-08-05T18:04:56.372112567Z Wrote inspect data to must-gather.
2026-08-05T18:04:56.401785933Z INFO: 'previous-logs' folder not found on ip-10-0-137-104.ec2.internal, skipping...
2026-08-05T18:04:56.401890534Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-08-05T18:04:56.401944662Z INFO: Machine Config Daemon termination log collection complete.
2026-08-05T18:04:56.430733499Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-08-05T18:04:56.444984920Z INFO: Getting status/config from prometheus-k8s-0
2026-08-05T18:04:56.484277216Z error: the server doesn't have a resource type "kubeletconfigs"
2026-08-05T18:04:56.646637485Z Wrote inspect data to must-gather.
2026-08-05T18:04:56.709030488Z Wrote inspect data to must-gather.
2026-08-05T18:04:56.860155713Z INFO: Getting status/flags from prometheus-k8s-0
2026-08-05T18:04:56.943189560Z Wrote inspect data to must-gather.
2026-08-05T18:04:56.962511164Z INFO: INTERCONNECT MODE
2026-08-05T18:04:56.962544367Z INFO: Gathering ovn-kubernetes DBs
2026-08-05T18:04:57.109599692Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-08-05T18:04:57.146482095Z INFO: Gathering OVN_Northbound from ovnkube-node-lxfxq...
2026-08-05T18:04:57.146731567Z INFO: Gathering OVN_Northbound from ovnkube-node-rd856...
2026-08-05T18:04:57.146988833Z INFO: Gathering OVN_Northbound from ovnkube-node-tsjgl...
2026-08-05T18:04:57.147345570Z INFO: Gathering OVN_Southbound from ovnkube-node-lxfxq...
2026-08-05T18:04:57.147619212Z INFO: Gathering OVN_Southbound from ovnkube-node-rd856...
2026-08-05T18:04:57.147882719Z INFO: Gathering OVN_Southbound from ovnkube-node-tsjgl...
2026-08-05T18:04:57.191804941Z Wrote inspect data to must-gather.
2026-08-05T18:04:57.371898996Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-08-05T18:04:57.507857867Z tar: Removing leading `/' from member names
2026-08-05T18:04:57.514693796Z tar: Removing leading `/' from member names
2026-08-05T18:04:57.532491745Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-08-05T18:04:57.542701925Z tar: Removing leading `/' from member names
2026-08-05T18:04:57.559658305Z tar: Removing leading `/' from member names
2026-08-05T18:04:57.567197809Z tar: Removing leading `/' from member names
2026-08-05T18:04:57.596270415Z Wrote inspect data to must-gather.
2026-08-05T18:04:57.599474014Z tar: Removing leading `/' from member names
2026-08-05T18:04:57.719744697Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-08-05T18:04:57.791101487Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-08-05T18:04:57.859470010Z Wrote inspect data to must-gather.
2026-08-05T18:04:57.969571010Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:58.099659612Z INFO: Waiting for network log collection to complete ...
2026-08-05T18:04:58.099685051Z INFO: Waiting for ovnk database copies to complete ...
2026-08-05T18:04:58.100090075Z INFO: Copying ovnk databases complete.
2026-08-05T18:04:58.100770605Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-08-05T18:04:58.102161436Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-08-05T18:04:58.104950439Z ovnk_database_store/
2026-08-05T18:04:58.104995496Z ovnk_database_store/ovnkube-node-lxfxq_nbdb
2026-08-05T18:04:58.118462567Z ovnk_database_store/ovnkube-node-rd856_nbdb
2026-08-05T18:04:58.127841066Z ovnk_database_store/ovnkube-node-rd856_sbdb
2026-08-05T18:04:58.151668468Z ovnk_database_store/ovnkube-node-tsjgl_nbdb
2026-08-05T18:04:58.161077179Z ovnk_database_store/ovnkube-node-lxfxq_sbdb
2026-08-05T18:04:58.186758076Z ovnk_database_store/ovnkube-node-tsjgl_sbdb
2026-08-05T18:04:58.216802307Z INFO: Network log collection complete.
2026-08-05T18:04:58.220876989Z Wrote inspect data to must-gather.
2026-08-05T18:04:58.221830224Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:58.224942660Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-08-05T18:04:58.244650703Z INFO: Getting status from alertmanager-main-0
2026-08-05T18:04:58.308252245Z Gathering data for ns/openshift-network-console...
2026-08-05T18:04:58.452205200Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:58.518610003Z Wrote inspect data to must-gather.
2026-08-05T18:04:58.662047341Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:58.665306049Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-08-05T18:04:58.778279119Z Gathering data for ns/openshift-console-operator...
2026-08-05T18:04:58.814888988Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-08-05T18:04:58.818005197Z INFO: Fallback to identify the container image from release info
2026-08-05T18:04:58.876631969Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:59.073320606Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:59.077967315Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-08-05T18:04:59.203302020Z Gathering data for ns/openshift-console...
2026-08-05T18:04:59.303679033Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:59.422306064Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2eb1033f55d7129b37ecfc3e4e9a306e42e5ab3dd7d2fe0ff08c8b45ea2b675
2026-08-05T18:04:59.509918240Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:59.514539373Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-08-05T18:04:59.571408960Z daemonset.apps/perf-node-gather-daemonset created
2026-08-05T18:04:59.709712890Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:59.736084239Z Waiting for performance profile collector pods to become ready: 1
2026-08-05T18:04:59.796396247Z Gathering data for ns/openshift-cluster-storage-operator...
2026-08-05T18:04:59.917794421Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:04:59.919970472Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-08-05T18:05:00.127920972Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:05:00.322845209Z Gathering data for ns/openshift-dns-operator...
2026-08-05T18:05:00.344034271Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:05:00.347502137Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-08-05T18:05:00.515610355Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:05:00.589982147Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T18:05:00.609106517Z Gathering data for ns/openshift-dns...
2026-08-05T18:05:00.698345606Z Wrote inspect data to /must-gather/istio.
2026-08-05T18:05:00.865710460Z Waiting for performance profile collector pods to become ready: 2
2026-08-05T18:05:01.214164861Z Gathering data for ns/openshift-image-registry...
2026-08-05T18:05:01.400096237Z Done executing Istio gather script
2026-08-05T18:05:01.676661761Z Gathering data for ns/openshift-ingress-operator...
2026-08-05T18:05:02.001625256Z Waiting for performance profile collector pods to become ready: 3
2026-08-05T18:05:02.036453314Z Gathering data for ns/openshift-ingress...
2026-08-05T18:05:02.341638833Z Gathering data for ns/openshift-ingress-canary...
2026-08-05T18:05:02.686038364Z Gathering data for ns/openshift-insights...
2026-08-05T18:05:03.122066595Z Waiting for performance profile collector pods to become ready: 4
2026-08-05T18:05:04.243140358Z Waiting for performance profile collector pods to become ready: 5
2026-08-05T18:05:04.974785383Z Gathering data for ns/kserve...
2026-08-05T18:05:05.373876064Z Waiting for performance profile collector pods to become ready: 6
2026-08-05T18:05:05.596676107Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T18:05:06.490710128Z Waiting for performance profile collector pods to become ready: 7
2026-08-05T18:05:06.531792764Z Gathering data for ns/openshift-config...
2026-08-05T18:05:06.780218728Z Gathering data for ns/openshift-config-managed...
2026-08-05T18:05:07.171228147Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-08-05T18:05:07.427987611Z Gathering data for ns/openshift-kube-apiserver...
2026-08-05T18:05:07.611313816Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-08-05T18:05:07.688794331Z Gathering data for ns/openshift-kube-controller-manager...
2026-08-05T18:05:07.933068409Z Collecting performance related data for node ip-10-0-135-2.ec2.internal
2026-08-05T18:05:07.941284794Z Collecting performance related data for node ip-10-0-135-121.ec2.internal
2026-08-05T18:05:07.952281095Z Collecting performance related data for node ip-10-0-137-104.ec2.internal
2026-08-05T18:05:08.033719687Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-08-05T18:05:08.392888696Z Gathering data for ns/openshift-kube-scheduler...
2026-08-05T18:05:08.688852437Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-08-05T18:05:09.040357480Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-08-05T18:05:09.410200469Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-08-05T18:05:09.746824539Z Gathering data for ns/openshift-user-workload-monitoring...
2026-08-05T18:05:10.258296617Z Gathering data for ns/openshift-multus...
2026-08-05T18:05:10.601261581Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T18:05:11.578993448Z Gathering data for ns/openshift-ovn-kubernetes...
2026-08-05T18:05:12.721986334Z Gathering data for ns/openshift-host-network...
2026-08-05T18:05:13.013543233Z Gathering data for ns/openshift-network-diagnostics...
2026-08-05T18:05:13.521939741Z Gathering data for ns/openshift-network-node-identity...
2026-08-05T18:05:13.926877283Z Gathering data for ns/openshift-network-operator...
2026-08-05T18:05:14.288024430Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-08-05T18:05:14.548768212Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-08-05T18:05:15.010671533Z Gathering data for ns/openshift-apiserver-operator...
2026-08-05T18:05:15.315505994Z Gathering data for ns/openshift-apiserver...
2026-08-05T18:05:15.607854958Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-08-05T18:05:15.702826189Z Gathering data for ns/openshift-controller-manager-operator...
2026-08-05T18:05:15.996687591Z Gathering data for ns/openshift-controller-manager...
2026-08-05T18:05:16.279477527Z Gathering data for ns/openshift-cluster-samples-operator...
2026-08-05T18:05:16.548745551Z Collecting kubelet logs for node ip-10-0-135-2.ec2.internal
2026-08-05T18:05:16.551469398Z Collecting kubelet logs for node ip-10-0-135-121.ec2.internal
2026-08-05T18:05:16.553566916Z Collecting kubelet logs for node ip-10-0-137-104.ec2.internal
2026-08-05T18:05:16.932813681Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-bm58p namespace
2026-08-05T18:05:16.937475757Z INFO: Node performance data collection complete.
2026-08-05T18:05:16.978845393Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-08-05T18:05:17.267229760Z Gathering data for ns/openshift-service-ca-operator...
2026-08-05T18:05:17.558909850Z Gathering data for ns/openshift-service-ca...
2026-08-05T18:05:17.906194896Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-08-05T18:05:18.732735386Z Wrote inspect data to must-gather.
2026-08-05T18:05:18.732779217Z error: inspection completed with the errors occurred while gathering data:
2026-08-05T18:05:18.732779217Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-08-05T18:05:18.861995802Z error: the server doesn't have a resource type "clusters"
2026-08-05T18:05:19.002627741Z Caches written to disk