2026-07-21T21:04:10.354394393Z [disk usage checker] Started
2026-07-21T21:04:10.358801773Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-21T21:04:10.560545016Z Gathering data for ns/openshift-cluster-version...
2026-07-21T21:04:10.716152312Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-21T21:04:10.809677802Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T21:04:10.892133340Z Gathering data for ns/default...
2026-07-21T21:04:11.222139952Z Gathering data for ns/openshift...
2026-07-21T21:04:11.684666149Z Gathering data for ns/kube-system...
2026-07-21T21:04:12.382209182Z Gathering data for ns/openshift-etcd...
2026-07-21T21:04:12.681149174Z Wrote inspect data to must-gather.
2026-07-21T21:04:12.681187817Z error: inspection completed with the errors occurred while gathering data:
2026-07-21T21:04:12.681187817Z     namespaces "assisted-installer" not found
2026-07-21T21:04:12.898940733Z Waiting on subprocesses to finish execution.
2026-07-21T21:04:12.914236811Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-21T21:04:12.915035149Z Executing Istio gather script
2026-07-21T21:04:12.918866642Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-21T21:04:12.920078826Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-21T21:04:12.931598246Z INFO: Gathering HAProxy config files
2026-07-21T21:04:12.955331434Z INFO: Waiting for node performance related collection to complete ...
2026-07-21T21:04:12.979576015Z INFO: Collecting host service logs for crio
2026-07-21T21:04:12.980026281Z INFO: Collecting host service logs for kubelet
2026-07-21T21:04:12.980402547Z INFO: Collecting host service logs for rpm-ostreed
2026-07-21T21:04:12.980740916Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-21T21:04:12.981081616Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-21T21:04:12.981404852Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-21T21:04:12.981710332Z INFO: Collecting host service logs for NetworkManager
2026-07-21T21:04:12.982095461Z INFO: Collecting host service logs for openvswitch
2026-07-21T21:04:12.982439912Z INFO: Collecting host service logs for ovs-configuration
2026-07-21T21:04:12.982774280Z INFO: Collecting host service logs for ovsdb-server
2026-07-21T21:04:12.983309016Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-21T21:04:12.983716819Z INFO: Waiting for worker host service log collection to complete ...
2026-07-21T21:04:13.351050060Z No resources found in openshift-etcd namespace.
2026-07-21T21:04:13.369780081Z INFO: "metallb-operator" not detected. Skipping.
2026-07-21T21:04:13.462462775Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-21T21:04:13.566123983Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-21T21:04:13.566220243Z INFO: on-disk MachineConfig config collection complete.
2026-07-21T21:04:13.727724592Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-21T21:04:13.792316484Z Wrote inspect data to must-gather.
2026-07-21T21:04:13.842446501Z No resources found
2026-07-21T21:04:13.852335227Z error: the server doesn't have a resource type "performanceprofile"
2026-07-21T21:04:14.010702421Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-21T21:04:14.222515721Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-pr7cp
2026-07-21T21:04:14.222558530Z insights-runtime-extractor-bv876
2026-07-21T21:04:14.222578277Z insights-runtime-extractor-dmtvw
2026-07-21T21:04:14.222597315Z insights-runtime-extractor-pk4b4
2026-07-21T21:04:14.288962321Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-21T21:04:14.633404641Z INFO: Worker host service log collection to complete.
2026-07-21T21:04:14.667968115Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T21:04:14.694782074Z Wrote inspect data to must-gather.
2026-07-21T21:04:14.787408950Z Wrote inspect data to must-gather.
2026-07-21T21:04:14.818303543Z Wrote inspect data to must-gather.
2026-07-21T21:04:14.905973290Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-21T21:04:14.921325699Z No resources found
2026-07-21T21:04:14.951112927Z ERROR: No running kube-apiserver pods found
2026-07-21T21:04:14.964277864Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-21T21:04:15.013600647Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-21T21:04:15.069145440Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-21T21:04:15.069145440Z See 'oc rsync -h' for help and examples
2026-07-21T21:04:15.095310523Z Wrote inspect data to must-gather.
2026-07-21T21:04:15.157142553Z Wrote inspect data to must-gather.
2026-07-21T21:04:15.195926066Z Wrote inspect data to must-gather.
2026-07-21T21:04:15.299151010Z INFO: 'previous-logs' folder not found on ip-10-0-134-27.ec2.internal, skipping...
2026-07-21T21:04:15.334225119Z tar: Removing leading `/' from member names
2026-07-21T21:04:15.339173562Z INFO: HAProxy config collection complete.
2026-07-21T21:04:15.357656468Z Wrote inspect data to must-gather.
2026-07-21T21:04:15.370572921Z error: the server doesn't have a resource type "machineconfigs"
2026-07-21T21:04:15.376932342Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-21T21:04:15.574312537Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-21T21:04:15.579843691Z Gathering data for ns/openshift-monitoring...
2026-07-21T21:04:15.583855883Z INFO: Getting rules from prometheus-k8s-0
2026-07-21T21:04:15.589698589Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-21T21:04:15.638052068Z INFO: 'previous-logs' folder not found on ip-10-0-135-100.ec2.internal, skipping...
2026-07-21T21:04:15.674577064Z Wrote inspect data to must-gather.
2026-07-21T21:04:15.815310593Z Wrote inspect data to must-gather.
2026-07-21T21:04:15.912389047Z INFO: 'previous-logs' folder not found on ip-10-0-140-146.ec2.internal, skipping...
2026-07-21T21:04:15.912494220Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-21T21:04:15.912550069Z INFO: Machine Config Daemon termination log collection complete.
2026-07-21T21:04:15.953701710Z Wrote inspect data to must-gather.
2026-07-21T21:04:15.980691807Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-21T21:04:16.025141298Z INFO: Getting status/config from prometheus-k8s-0
2026-07-21T21:04:16.204629115Z Wrote inspect data to must-gather.
2026-07-21T21:04:16.229699406Z INFO: INTERCONNECT MODE
2026-07-21T21:04:16.229983319Z INFO: Gathering ovn-kubernetes DBs
2026-07-21T21:04:16.315068147Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-21T21:04:16.397721363Z INFO: Gathering OVN_Northbound from ovnkube-node-96t4c...
2026-07-21T21:04:16.397992486Z INFO: Gathering OVN_Northbound from ovnkube-node-9c6cc...
2026-07-21T21:04:16.398261138Z INFO: Gathering OVN_Northbound from ovnkube-node-9sjnv...
2026-07-21T21:04:16.398746449Z INFO: Gathering OVN_Southbound from ovnkube-node-96t4c...
2026-07-21T21:04:16.399028338Z INFO: Gathering OVN_Southbound from ovnkube-node-9c6cc...
2026-07-21T21:04:16.399324679Z INFO: Gathering OVN_Southbound from ovnkube-node-9sjnv...
2026-07-21T21:04:16.410373816Z Wrote inspect data to must-gather.
2026-07-21T21:04:16.478097250Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-21T21:04:16.565448289Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:16.690384475Z tar: Removing leading `/' from member names
2026-07-21T21:04:16.709293835Z tar: Removing leading `/' from member names
2026-07-21T21:04:16.710053339Z tar: Removing leading `/' from member names
2026-07-21T21:04:16.732459309Z tar: Removing leading `/' from member names
2026-07-21T21:04:16.735933466Z tar: Removing leading `/' from member names
2026-07-21T21:04:16.739812008Z tar: Removing leading `/' from member names
2026-07-21T21:04:16.773668122Z Wrote inspect data to must-gather.
2026-07-21T21:04:16.840182911Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-21T21:04:16.844336108Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:16.847400735Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-21T21:04:17.004527928Z Wrote inspect data to must-gather.
2026-07-21T21:04:17.083007535Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:17.121277844Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-21T21:04:17.234222863Z INFO: Waiting for network log collection to complete ...
2026-07-21T21:04:17.234250691Z INFO: Waiting for ovnk database copies to complete ...
2026-07-21T21:04:17.234647620Z INFO: Copying ovnk databases complete.
2026-07-21T21:04:17.236599403Z 2.4M	must-gather/network_logs/ovnk_database_store
2026-07-21T21:04:17.236986229Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-21T21:04:17.239747516Z ovnk_database_store/
2026-07-21T21:04:17.239819909Z ovnk_database_store/ovnkube-node-96t4c_sbdb
2026-07-21T21:04:17.244905331Z Wrote inspect data to must-gather.
2026-07-21T21:04:17.257046296Z ovnk_database_store/ovnkube-node-9c6cc_nbdb
2026-07-21T21:04:17.263220101Z ovnk_database_store/ovnkube-node-9c6cc_sbdb
2026-07-21T21:04:17.277315006Z ovnk_database_store/ovnkube-node-9sjnv_nbdb
2026-07-21T21:04:17.283168741Z ovnk_database_store/ovnkube-node-96t4c_nbdb
2026-07-21T21:04:17.289452125Z ovnk_database_store/ovnkube-node-9sjnv_sbdb
2026-07-21T21:04:17.309383331Z INFO: Network log collection complete.
2026-07-21T21:04:17.332646902Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:17.336226695Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-21T21:04:17.429222160Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-21T21:04:17.472087301Z Wrote inspect data to must-gather.
2026-07-21T21:04:17.556679264Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:17.679377491Z Wrote inspect data to must-gather.
2026-07-21T21:04:17.761600105Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:17.764625656Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-21T21:04:17.821682379Z INFO: Getting status from alertmanager-main-0
2026-07-21T21:04:17.863456006Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T21:04:17.949893026Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-21T21:04:17.952146553Z INFO: Fallback to identify the container image from release info
2026-07-21T21:04:18.006082838Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:18.044719712Z Gathering data for ns/openshift-network-console...
2026-07-21T21:04:18.194881552Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:18.197063068Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-21T21:04:18.343403242Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:18.446813415Z Gathering data for ns/openshift-console-operator...
2026-07-21T21:04:18.525177662Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:04:18.789292629Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-21T21:04:18.833815387Z Gathering data for ns/openshift-console...
2026-07-21T21:04:18.922936018Z daemonset.apps/perf-node-gather-daemonset created
2026-07-21T21:04:19.047404783Z Waiting for performance profile collector pods to become ready: 1
2026-07-21T21:04:19.210136102Z Done executing Istio gather script
2026-07-21T21:04:19.415678590Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-21T21:04:19.915056227Z Gathering data for ns/openshift-dns-operator...
2026-07-21T21:04:20.168694511Z Waiting for performance profile collector pods to become ready: 2
2026-07-21T21:04:20.200745312Z Gathering data for ns/openshift-dns...
2026-07-21T21:04:20.382308967Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-21T21:04:20.789822464Z Gathering data for ns/openshift-image-registry...
2026-07-21T21:04:21.243852660Z Gathering data for ns/openshift-ingress-operator...
2026-07-21T21:04:21.290224614Z Waiting for performance profile collector pods to become ready: 3
2026-07-21T21:04:21.603228012Z Gathering data for ns/openshift-ingress...
2026-07-21T21:04:21.943851489Z Gathering data for ns/openshift-ingress-canary...
2026-07-21T21:04:22.319261163Z Gathering data for ns/openshift-insights...
2026-07-21T21:04:22.394753981Z Waiting for performance profile collector pods to become ready: 4
2026-07-21T21:04:23.500492765Z Waiting for performance profile collector pods to become ready: 5
2026-07-21T21:04:24.111657742Z Gathering data for ns/openshift-config...
2026-07-21T21:04:24.374904374Z Gathering data for ns/openshift-config-managed...
2026-07-21T21:04:24.611378195Z Waiting for performance profile collector pods to become ready: 6
2026-07-21T21:04:24.794201464Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-21T21:04:25.041062261Z Gathering data for ns/openshift-kube-apiserver...
2026-07-21T21:04:25.281726471Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-21T21:04:25.387058332Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-21T21:04:25.570610756Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-21T21:04:25.718158445Z Waiting for performance profile collector pods to become ready: 7
2026-07-21T21:04:25.884155560Z Gathering data for ns/openshift-kube-scheduler...
2026-07-21T21:04:26.162562122Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-21T21:04:26.473209754Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-21T21:04:26.819504310Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-21T21:04:26.832706589Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-21T21:04:27.070678770Z Collecting performance related data for node ip-10-0-134-27.ec2.internal
2026-07-21T21:04:27.079917945Z Collecting performance related data for node ip-10-0-140-146.ec2.internal
2026-07-21T21:04:27.090803658Z Collecting performance related data for node ip-10-0-135-100.ec2.internal
2026-07-21T21:04:27.175178475Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-21T21:04:27.759491325Z Gathering data for ns/openshift-multus...
2026-07-21T21:04:29.396138589Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-21T21:04:30.392145139Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-21T21:04:30.632446561Z Gathering data for ns/openshift-host-network...
2026-07-21T21:04:30.976871733Z Gathering data for ns/openshift-network-diagnostics...
2026-07-21T21:04:31.524755688Z Gathering data for ns/openshift-network-node-identity...
2026-07-21T21:04:31.930237073Z Gathering data for ns/openshift-network-operator...
2026-07-21T21:04:32.305875347Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-21T21:04:32.652219102Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-21T21:04:33.135439219Z Gathering data for ns/openshift-apiserver-operator...
2026-07-21T21:04:33.415712056Z Gathering data for ns/openshift-apiserver...
2026-07-21T21:04:33.785278086Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-21T21:04:34.040861868Z Gathering data for ns/openshift-controller-manager...
2026-07-21T21:04:34.324516386Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-21T21:04:35.004047113Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-21T21:04:35.024126135Z Collecting kubelet logs for node ip-10-0-134-27.ec2.internal
2026-07-21T21:04:35.026494007Z Collecting kubelet logs for node ip-10-0-140-146.ec2.internal
2026-07-21T21:04:35.028887950Z Collecting kubelet logs for node ip-10-0-135-100.ec2.internal
2026-07-21T21:04:35.319364859Z Gathering data for ns/openshift-service-ca-operator...
2026-07-21T21:04:35.376024184Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8dk8j namespace
2026-07-21T21:04:35.382758129Z INFO: Node performance data collection complete.
2026-07-21T21:04:35.408745939Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-21T21:04:35.651196828Z Gathering data for ns/openshift-service-ca...
2026-07-21T21:04:36.008997765Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-21T21:04:36.850909241Z Wrote inspect data to must-gather.
2026-07-21T21:04:36.850951432Z error: inspection completed with the errors occurred while gathering data:
2026-07-21T21:04:36.850951432Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-21T21:04:36.972131838Z error: the server doesn't have a resource type "clusters"
2026-07-21T21:04:37.092744962Z Caches written to disk