2026-07-21T17:25:03.579003821Z [disk usage checker] Started
2026-07-21T17:25:03.582303781Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T17:25:03.776254992Z Gathering data for ns/openshift-cluster-version...
2026-07-21T17:25:03.914267719Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-21T17:25:04.009111258Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T17:25:04.100773915Z Gathering data for ns/default...
2026-07-21T17:25:04.377502267Z Gathering data for ns/openshift...
2026-07-21T17:25:04.765711938Z Gathering data for ns/kube-system...
2026-07-21T17:25:05.402368846Z Gathering data for ns/openshift-etcd...
2026-07-21T17:25:05.685120229Z Wrote inspect data to must-gather.
2026-07-21T17:25:05.685155972Z error: inspection completed with the errors occurred while gathering data:
2026-07-21T17:25:05.685155972Z     namespaces "assisted-installer" not found
2026-07-21T17:25:06.135457274Z Waiting on subprocesses to finish execution.
2026-07-21T17:25:06.141687346Z INFO: Gathering HAProxy config files
2026-07-21T17:25:06.151249867Z INFO: Collecting host service logs for crio
2026-07-21T17:25:06.151768801Z INFO: Collecting host service logs for kubelet
2026-07-21T17:25:06.151903195Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-21T17:25:06.152129521Z INFO: Collecting host service logs for rpm-ostreed
2026-07-21T17:25:06.152479692Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-21T17:25:06.152874335Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-21T17:25:06.153196159Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-21T17:25:06.153537681Z INFO: Collecting host service logs for NetworkManager
2026-07-21T17:25:06.153888191Z INFO: Collecting host service logs for openvswitch
2026-07-21T17:25:06.154229349Z INFO: Collecting host service logs for ovs-configuration
2026-07-21T17:25:06.154584014Z INFO: Collecting host service logs for ovsdb-server
2026-07-21T17:25:06.154927668Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-21T17:25:06.155042315Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-21T17:25:06.155334408Z INFO: Waiting for worker host service log collection to complete ...
2026-07-21T17:25:06.167374205Z Executing Istio gather script
2026-07-21T17:25:06.257938127Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-21T17:25:06.488100120Z INFO: Waiting for node performance related collection to complete ...
2026-07-21T17:25:06.525566780Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-21T17:25:06.525732486Z INFO: on-disk MachineConfig config collection complete.
2026-07-21T17:25:07.178724229Z Wrote inspect data to must-gather.
2026-07-21T17:25:07.228388914Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-c58ls
2026-07-21T17:25:07.228447891Z insights-runtime-extractor-2nvpt
2026-07-21T17:25:07.228470701Z insights-runtime-extractor-97vdj
2026-07-21T17:25:07.228509309Z insights-runtime-extractor-djqmh
2026-07-21T17:25:07.230871297Z No resources found
2026-07-21T17:25:07.310438879Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-21T17:25:07.320241972Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-21T17:25:07.340754521Z No resources found in openshift-etcd namespace.
2026-07-21T17:25:07.358278582Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-21T17:25:07.420568855Z INFO: "metallb-operator" not detected. Skipping.
2026-07-21T17:25:07.469986477Z error: the server doesn't have a resource type "performanceprofile"
2026-07-21T17:25:07.519287476Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-21T17:25:07.564707179Z INFO: Worker host service log collection to complete.
2026-07-21T17:25:07.641134505Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-21T17:25:07.641134505Z See 'oc rsync -h' for help and examples
2026-07-21T17:25:07.708251760Z ERROR: No running kube-apiserver pods found
2026-07-21T17:25:07.885857271Z Wrote inspect data to must-gather.
2026-07-21T17:25:08.130620447Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-21T17:25:08.190264054Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-21T17:25:08.362207158Z Gathering data for ns/openshift-monitoring...
2026-07-21T17:25:08.396176283Z No resources found
2026-07-21T17:25:08.484737721Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T17:25:08.513875668Z Wrote inspect data to must-gather.
2026-07-21T17:25:08.585701257Z Wrote inspect data to must-gather.
2026-07-21T17:25:08.586303492Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-21T17:25:08.627920708Z Wrote inspect data to must-gather.
2026-07-21T17:25:08.646318471Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T17:25:08.672864295Z Wrote inspect data to must-gather.
2026-07-21T17:25:08.697326766Z Wrote inspect data to must-gather.
2026-07-21T17:25:08.711678972Z INFO: 'previous-logs' folder not found on ip-10-0-136-132.ec2.internal, skipping...
2026-07-21T17:25:08.878694567Z tar: Removing leading `/' from member names
2026-07-21T17:25:08.882408786Z INFO: HAProxy config collection complete.
2026-07-21T17:25:08.901914647Z error: the server doesn't have a resource type "machineconfigs"
2026-07-21T17:25:08.946416806Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-21T17:25:08.966385192Z Wrote inspect data to must-gather.
2026-07-21T17:25:09.024705071Z INFO: 'previous-logs' folder not found on ip-10-0-137-79.ec2.internal, skipping...
2026-07-21T17:25:09.074933551Z INFO: Getting rules from prometheus-k8s-0
2026-07-21T17:25:09.081183869Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-21T17:25:09.241378961Z Wrote inspect data to must-gather.
2026-07-21T17:25:09.306045915Z INFO: 'previous-logs' folder not found on ip-10-0-139-58.ec2.internal, skipping...
2026-07-21T17:25:09.306077917Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-21T17:25:09.306110414Z INFO: Machine Config Daemon termination log collection complete.
2026-07-21T17:25:09.319545974Z Wrote inspect data to must-gather.
2026-07-21T17:25:09.470177138Z INFO: Getting status/config from prometheus-k8s-0
2026-07-21T17:25:09.488985823Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-21T17:25:09.496283598Z Wrote inspect data to must-gather.
2026-07-21T17:25:09.726866418Z Wrote inspect data to must-gather.
2026-07-21T17:25:09.791565310Z INFO: INTERCONNECT MODE
2026-07-21T17:25:09.791604628Z INFO: Gathering ovn-kubernetes DBs
2026-07-21T17:25:09.844927819Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-21T17:25:09.865787293Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-21T17:25:09.955674683Z Wrote inspect data to must-gather.
2026-07-21T17:25:09.969325433Z INFO: Gathering OVN_Northbound from ovnkube-node-6bvgd...
2026-07-21T17:25:09.969589096Z INFO: Gathering OVN_Northbound from ovnkube-node-m5cjh...
2026-07-21T17:25:09.969837073Z INFO: Gathering OVN_Northbound from ovnkube-node-vz6h9...
2026-07-21T17:25:09.970302905Z INFO: Gathering OVN_Southbound from ovnkube-node-6bvgd...
2026-07-21T17:25:09.970616341Z INFO: Gathering OVN_Southbound from ovnkube-node-m5cjh...
2026-07-21T17:25:09.970902068Z INFO: Gathering OVN_Southbound from ovnkube-node-vz6h9...
2026-07-21T17:25:10.180942253Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-21T17:25:10.220711387Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-21T17:25:10.262679611Z tar: Removing leading `/' from member names
2026-07-21T17:25:10.329687858Z tar: Removing leading `/' from member names
2026-07-21T17:25:10.335097618Z tar: Removing leading `/' from member names
2026-07-21T17:25:10.355735776Z tar: Removing leading `/' from member names
2026-07-21T17:25:10.362088629Z tar: Removing leading `/' from member names
2026-07-21T17:25:10.378537591Z tar: Removing leading `/' from member names
2026-07-21T17:25:10.403069576Z Wrote inspect data to must-gather.
2026-07-21T17:25:10.490110658Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:10.508784023Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-21T17:25:10.676128913Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T17:25:10.758217842Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:10.761742539Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-21T17:25:10.766872996Z Wrote inspect data to must-gather.
2026-07-21T17:25:10.779157573Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-21T17:25:10.881978821Z INFO: Waiting for network log collection to complete ...
2026-07-21T17:25:10.882139138Z INFO: Waiting for ovnk database copies to complete ...
2026-07-21T17:25:10.882735174Z INFO: Copying ovnk databases complete.
2026-07-21T17:25:10.886242246Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-21T17:25:10.889076202Z ovnk_database_store/
2026-07-21T17:25:10.889076202Z ovnk_database_store/ovnkube-node-vz6h9_nbdb
2026-07-21T17:25:10.889229058Z Gathering data for ns/openshift-network-console...
2026-07-21T17:25:10.901586131Z ovnk_database_store/ovnkube-node-6bvgd_nbdb
2026-07-21T17:25:10.907559949Z ovnk_database_store/ovnkube-node-vz6h9_sbdb
2026-07-21T17:25:10.926223165Z ovnk_database_store/ovnkube-node-m5cjh_sbdb
2026-07-21T17:25:10.947718110Z ovnk_database_store/ovnkube-node-6bvgd_sbdb
2026-07-21T17:25:10.969699305Z ovnk_database_store/ovnkube-node-m5cjh_nbdb
2026-07-21T17:25:10.980424379Z INFO: Network log collection complete.
2026-07-21T17:25:11.000349568Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:11.116153223Z Wrote inspect data to must-gather.
2026-07-21T17:25:11.175237079Z INFO: Getting status from alertmanager-main-0
2026-07-21T17:25:11.228291718Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:11.232589312Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-21T17:25:11.379012613Z Gathering data for ns/openshift-console-operator...
2026-07-21T17:25:11.436874203Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:11.457964201Z Wrote inspect data to must-gather.
2026-07-21T17:25:11.651552943Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:11.653982458Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-21T17:25:11.767612050Z Gathering data for ns/openshift-console...
2026-07-21T17:25:11.778822552Z Wrote inspect data to must-gather.
2026-07-21T17:25:11.873871794Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:12.017637044Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-21T17:25:12.021309537Z INFO: Fallback to identify the container image from release info
2026-07-21T17:25:12.075769928Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:12.078306446Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-21T17:25:12.252136205Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:12.300828837Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-21T17:25:12.446879425Z Wrote inspect data to /must-gather/istio.
2026-07-21T17:25:12.715139412Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-21T17:25:12.794818839Z Gathering data for ns/openshift-dns-operator...
2026-07-21T17:25:12.854943125Z daemonset.apps/perf-node-gather-daemonset created
2026-07-21T17:25:12.983055137Z Waiting for performance profile collector pods to become ready: 1
2026-07-21T17:25:13.095638049Z Gathering data for ns/openshift-dns...
2026-07-21T17:25:13.137322239Z Done executing Istio gather script
2026-07-21T17:25:13.661507076Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T17:25:13.683409164Z Gathering data for ns/openshift-image-registry...
2026-07-21T17:25:14.102692006Z Waiting for performance profile collector pods to become ready: 2
2026-07-21T17:25:14.104751829Z Gathering data for ns/openshift-ingress-operator...
2026-07-21T17:25:14.392722895Z Gathering data for ns/openshift-ingress...
2026-07-21T17:25:14.677976600Z Gathering data for ns/openshift-ingress-canary...
2026-07-21T17:25:15.023830707Z Gathering data for ns/openshift-insights...
2026-07-21T17:25:15.203508402Z Waiting for performance profile collector pods to become ready: 3
2026-07-21T17:25:16.309733025Z Waiting for performance profile collector pods to become ready: 4
2026-07-21T17:25:16.785945341Z Gathering data for ns/openshift-jobset-operator...
2026-07-21T17:25:17.417569281Z Waiting for performance profile collector pods to become ready: 5
2026-07-21T17:25:17.540535170Z Gathering data for ns/openshift-config...
2026-07-21T17:25:17.776092417Z Gathering data for ns/openshift-config-managed...
2026-07-21T17:25:18.152227198Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-21T17:25:18.379744745Z Gathering data for ns/openshift-kube-apiserver...
2026-07-21T17:25:18.530326780Z Waiting for performance profile collector pods to become ready: 6
2026-07-21T17:25:18.623227805Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-21T17:25:18.668221160Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T17:25:18.876232256Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-21T17:25:19.138916199Z Gathering data for ns/openshift-kube-scheduler...
2026-07-21T17:25:19.371705383Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-21T17:25:19.654919636Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-21T17:25:19.657649216Z Waiting for performance profile collector pods to become ready: 7
2026-07-21T17:25:19.944474858Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-21T17:25:20.224542732Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-21T17:25:20.724218749Z Gathering data for ns/openshift-multus...
2026-07-21T17:25:20.772298076Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-21T17:25:20.987954803Z Collecting performance related data for node ip-10-0-139-58.ec2.internal
2026-07-21T17:25:20.995160394Z Collecting performance related data for node ip-10-0-136-132.ec2.internal
2026-07-21T17:25:21.003036155Z Collecting performance related data for node ip-10-0-137-79.ec2.internal
2026-07-21T17:25:22.205235529Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-21T17:25:23.400941943Z Gathering data for ns/openshift-host-network...
2026-07-21T17:25:23.674662745Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T17:25:23.692994880Z Gathering data for ns/openshift-network-diagnostics...
2026-07-21T17:25:24.203365231Z Gathering data for ns/openshift-network-node-identity...
2026-07-21T17:25:24.565666354Z Gathering data for ns/openshift-network-operator...
2026-07-21T17:25:24.924603443Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-21T17:25:25.170868657Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-21T17:25:25.635747289Z Gathering data for ns/openshift-apiserver-operator...
2026-07-21T17:25:25.953002636Z Gathering data for ns/openshift-apiserver...
2026-07-21T17:25:26.357076418Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-21T17:25:26.602296523Z Gathering data for ns/openshift-controller-manager...
2026-07-21T17:25:26.849792861Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-21T17:25:27.505716005Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-21T17:25:27.779711196Z Gathering data for ns/openshift-service-ca-operator...
2026-07-21T17:25:28.073896622Z Gathering data for ns/openshift-service-ca...
2026-07-21T17:25:28.400543660Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-21T17:25:28.680252537Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T17:25:29.253504284Z Wrote inspect data to must-gather.
2026-07-21T17:25:29.253543292Z error: inspection completed with the errors occurred while gathering data:
2026-07-21T17:25:29.253543292Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-21T17:25:29.268566093Z Collecting kubelet logs for node ip-10-0-139-58.ec2.internal
2026-07-21T17:25:29.270523587Z Collecting kubelet logs for node ip-10-0-136-132.ec2.internal
2026-07-21T17:25:29.273012849Z Collecting kubelet logs for node ip-10-0-137-79.ec2.internal
2026-07-21T17:25:29.559472415Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-ckmhs namespace
2026-07-21T17:25:29.564283183Z INFO: Node performance data collection complete.
2026-07-21T17:25:29.693423205Z error: the server doesn't have a resource type "clusters"
2026-07-21T17:25:29.820820940Z Caches written to disk