2026-07-13T21:08:47.776888960Z [disk usage checker] Started
2026-07-13T21:08:47.780575581Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-13T21:08:47.975670142Z Gathering data for ns/openshift-cluster-version...
2026-07-13T21:08:48.102086408Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-13T21:08:48.198501528Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T21:08:48.280961385Z Gathering data for ns/default...
2026-07-13T21:08:48.571713521Z Gathering data for ns/openshift...
2026-07-13T21:08:48.980054635Z Gathering data for ns/kube-system...
2026-07-13T21:08:49.660949459Z Gathering data for ns/openshift-etcd...
2026-07-13T21:08:49.941731350Z Wrote inspect data to must-gather.
2026-07-13T21:08:49.941764531Z error: inspection completed with the errors occurred while gathering data:
2026-07-13T21:08:49.941764531Z     namespaces "assisted-installer" not found
2026-07-13T21:08:50.522237960Z Waiting on subprocesses to finish execution.
2026-07-13T21:08:50.531368879Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-13T21:08:50.534016102Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-13T21:08:50.549879325Z Executing Istio gather script
2026-07-13T21:08:50.553442098Z INFO: Collecting host service logs for crio
2026-07-13T21:08:50.553786623Z INFO: Collecting host service logs for kubelet
2026-07-13T21:08:50.554231088Z INFO: Collecting host service logs for rpm-ostreed
2026-07-13T21:08:50.554561895Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-13T21:08:50.554893215Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-13T21:08:50.555252124Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-13T21:08:50.555594568Z INFO: Collecting host service logs for NetworkManager
2026-07-13T21:08:50.555942630Z INFO: Collecting host service logs for openvswitch
2026-07-13T21:08:50.556296677Z INFO: Collecting host service logs for ovs-configuration
2026-07-13T21:08:50.556675912Z INFO: Collecting host service logs for ovsdb-server
2026-07-13T21:08:50.557228252Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-13T21:08:50.557638526Z INFO: Waiting for worker host service log collection to complete ...
2026-07-13T21:08:50.560293419Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-13T21:08:50.571793352Z INFO: Gathering HAProxy config files
2026-07-13T21:08:50.590615206Z INFO: Waiting for node performance related collection to complete ...
2026-07-13T21:08:50.838877873Z No resources found in openshift-etcd namespace.
2026-07-13T21:08:50.989555087Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-t45cw
2026-07-13T21:08:50.989702592Z insights-runtime-extractor-7p5kz
2026-07-13T21:08:50.989736774Z insights-runtime-extractor-kbtzv
2026-07-13T21:08:50.989767360Z insights-runtime-extractor-ngz5s
2026-07-13T21:08:51.268333520Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-13T21:08:51.285309876Z Wrote inspect data to must-gather.
2026-07-13T21:08:51.385891548Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-13T21:08:51.385891548Z See 'oc rsync -h' for help and examples
2026-07-13T21:08:51.516619958Z No resources found
2026-07-13T21:08:51.577453722Z INFO: "metallb-operator" not detected. Skipping.
2026-07-13T21:08:51.650530934Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-13T21:08:51.855952323Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-13T21:08:51.947217679Z ERROR: No running kube-apiserver pods found
2026-07-13T21:08:51.967577325Z error: the server doesn't have a resource type "performanceprofile"
2026-07-13T21:08:52.037885342Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-13T21:08:52.037974907Z INFO: on-disk MachineConfig config collection complete.
2026-07-13T21:08:52.111562859Z INFO: Worker host service log collection to complete.
2026-07-13T21:08:52.387439907Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-13T21:08:52.392746883Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-13T21:08:52.635518234Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T21:08:52.680494320Z No resources found
2026-07-13T21:08:52.777253075Z Wrote inspect data to must-gather.
2026-07-13T21:08:52.827748654Z Wrote inspect data to must-gather.
2026-07-13T21:08:52.839941593Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-13T21:08:52.898578051Z Wrote inspect data to must-gather.
2026-07-13T21:08:52.970381896Z Gathering data for ns/openshift-monitoring...
2026-07-13T21:08:53.023153104Z tar: Removing leading `/' from member names
2026-07-13T21:08:53.028479297Z Wrote inspect data to must-gather.
2026-07-13T21:08:53.029673695Z INFO: HAProxy config collection complete.
2026-07-13T21:08:53.066022743Z Wrote inspect data to must-gather.
2026-07-13T21:08:53.067080021Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-13T21:08:53.116212036Z Wrote inspect data to must-gather.
2026-07-13T21:08:53.120142689Z INFO: 'previous-logs' folder not found on ip-10-0-128-233.ec2.internal, skipping...
2026-07-13T21:08:53.200288036Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-13T21:08:53.252735182Z error: the server doesn't have a resource type "machineconfigs"
2026-07-13T21:08:53.294856098Z Wrote inspect data to must-gather.
2026-07-13T21:08:53.472862013Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-13T21:08:53.473724985Z INFO: 'previous-logs' folder not found on ip-10-0-128-52.ec2.internal, skipping...
2026-07-13T21:08:53.596747835Z Wrote inspect data to must-gather.
2026-07-13T21:08:53.742269775Z Wrote inspect data to must-gather.
2026-07-13T21:08:53.772371508Z INFO: Getting rules from prometheus-k8s-0
2026-07-13T21:08:53.779981745Z INFO: 'previous-logs' folder not found on ip-10-0-134-97.ec2.internal, skipping...
2026-07-13T21:08:53.780050316Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-13T21:08:53.780082757Z INFO: Machine Config Daemon termination log collection complete.
2026-07-13T21:08:53.921181030Z Wrote inspect data to must-gather.
2026-07-13T21:08:53.939218845Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-13T21:08:54.196196334Z Wrote inspect data to must-gather.
2026-07-13T21:08:54.222682748Z INFO: Getting status/config from prometheus-k8s-0
2026-07-13T21:08:54.249552169Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-13T21:08:54.273236172Z INFO: INTERCONNECT MODE
2026-07-13T21:08:54.273272202Z INFO: Gathering ovn-kubernetes DBs
2026-07-13T21:08:54.467971956Z Wrote inspect data to must-gather.
2026-07-13T21:08:54.468591176Z INFO: Gathering OVN_Northbound from ovnkube-node-4sksc...
2026-07-13T21:08:54.468877206Z INFO: Gathering OVN_Northbound from ovnkube-node-c9wsz...
2026-07-13T21:08:54.469224103Z INFO: Gathering OVN_Northbound from ovnkube-node-cwl74...
2026-07-13T21:08:54.469718269Z INFO: Gathering OVN_Southbound from ovnkube-node-4sksc...
2026-07-13T21:08:54.470060077Z INFO: Gathering OVN_Southbound from ovnkube-node-c9wsz...
2026-07-13T21:08:54.470455513Z INFO: Gathering OVN_Southbound from ovnkube-node-cwl74...
2026-07-13T21:08:54.638977729Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-13T21:08:54.778238519Z tar: Removing leading `/' from member names
2026-07-13T21:08:54.833829533Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-13T21:08:54.856290628Z tar: Removing leading `/' from member names
2026-07-13T21:08:54.857441624Z tar: Removing leading `/' from member names
2026-07-13T21:08:54.863182808Z tar: Removing leading `/' from member names
2026-07-13T21:08:54.870160695Z tar: Removing leading `/' from member names
2026-07-13T21:08:54.889932830Z tar: Removing leading `/' from member names
2026-07-13T21:08:54.928620891Z Wrote inspect data to must-gather.
2026-07-13T21:08:55.121072520Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-13T21:08:55.185783947Z Wrote inspect data to must-gather.
2026-07-13T21:08:55.211162861Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-13T21:08:55.388688898Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-13T21:08:55.390901448Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-13T21:08:55.423495279Z INFO: Waiting for network log collection to complete ...
2026-07-13T21:08:55.423611691Z INFO: Waiting for ovnk database copies to complete ...
2026-07-13T21:08:55.424191019Z INFO: Copying ovnk databases complete.
2026-07-13T21:08:55.424653102Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:55.425948199Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-13T21:08:55.428471246Z ovnk_database_store/
2026-07-13T21:08:55.428471246Z ovnk_database_store/ovnkube-node-c9wsz_nbdb
2026-07-13T21:08:55.441585466Z ovnk_database_store/ovnkube-node-4sksc_nbdb
2026-07-13T21:08:55.453283458Z ovnk_database_store/ovnkube-node-4sksc_sbdb
2026-07-13T21:08:55.479075653Z ovnk_database_store/ovnkube-node-c9wsz_sbdb
2026-07-13T21:08:55.504027003Z ovnk_database_store/ovnkube-node-cwl74_nbdb
2026-07-13T21:08:55.514239328Z ovnk_database_store/ovnkube-node-cwl74_sbdb
2026-07-13T21:08:55.514422733Z Wrote inspect data to must-gather.
2026-07-13T21:08:55.540462750Z INFO: Network log collection complete.
2026-07-13T21:08:55.614447992Z Gathering data for ns/openshift-network-console...
2026-07-13T21:08:55.658922995Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-13T21:08:55.659551554Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:55.665243392Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-13T21:08:55.831743530Z Wrote inspect data to must-gather.
2026-07-13T21:08:55.877502643Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:56.075808161Z INFO: Getting status from alertmanager-main-0
2026-07-13T21:08:56.130211559Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:56.133603941Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-13T21:08:56.150151185Z Gathering data for ns/openshift-console-operator...
2026-07-13T21:08:56.158977758Z Wrote inspect data to must-gather.
2026-07-13T21:08:56.355469189Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:56.447407204Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-13T21:08:56.450373787Z INFO: Fallback to identify the container image from release info
2026-07-13T21:08:56.581191286Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:56.584728924Z Gathering data for ns/openshift-console...
2026-07-13T21:08:56.584933225Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-13T21:08:56.800244301Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:57.001413937Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:57.004554550Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-13T21:08:57.100196531Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-13T21:08:57.202816023Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:57.240842604Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-13T21:08:57.365433361Z daemonset.apps/perf-node-gather-daemonset created
2026-07-13T21:08:57.418348166Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:57.422661488Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-13T21:08:57.525941711Z Waiting for performance profile collector pods to become ready: 1
2026-07-13T21:08:57.632092845Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:57.653958065Z Gathering data for ns/openshift-dns-operator...
2026-07-13T21:08:57.841651600Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:57.845182223Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-13T21:08:57.855271911Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-13T21:08:57.997279668Z Gathering data for ns/openshift-dns...
2026-07-13T21:08:58.035371400Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:58.208353671Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:58.210782562Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-13T21:08:58.380952938Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:58.580095377Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:58.583675634Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-13T21:08:58.678454378Z Waiting for performance profile collector pods to become ready: 2
2026-07-13T21:08:58.731535432Z Gathering data for ns/openshift-image-registry...
2026-07-13T21:08:58.774298324Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:58.963468992Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:58.965595247Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-13T21:08:59.130893093Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:59.249521550Z Gathering data for ns/openshift-ingress-operator...
2026-07-13T21:08:59.339669810Z Wrote inspect data to /must-gather/istio.
2026-07-13T21:08:59.591706752Z Gathering data for ns/openshift-ingress...
2026-07-13T21:08:59.814274485Z Waiting for performance profile collector pods to become ready: 3
2026-07-13T21:08:59.943793141Z Gathering data for ns/openshift-ingress-canary...
2026-07-13T21:09:00.056181467Z Done executing Istio gather script
2026-07-13T21:09:00.318360806Z Gathering data for ns/openshift-insights...
2026-07-13T21:09:00.946221682Z Waiting for performance profile collector pods to become ready: 4
2026-07-13T21:09:02.063347212Z Waiting for performance profile collector pods to become ready: 5
2026-07-13T21:09:02.591242583Z Gathering data for ns/kserve...
2026-07-13T21:09:02.859967312Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-13T21:09:03.184466794Z Waiting for performance profile collector pods to become ready: 6
2026-07-13T21:09:04.306930878Z Waiting for performance profile collector pods to become ready: 7
2026-07-13T21:09:04.347118960Z Gathering data for ns/openshift-config...
2026-07-13T21:09:04.607178620Z Gathering data for ns/openshift-config-managed...
2026-07-13T21:09:05.059740724Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-13T21:09:05.319736065Z Gathering data for ns/openshift-kube-apiserver...
2026-07-13T21:09:05.438664430Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-13T21:09:05.631644527Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-13T21:09:05.726207298Z Collecting performance related data for node ip-10-0-128-233.ec2.internal
2026-07-13T21:09:05.733334961Z Collecting performance related data for node ip-10-0-128-52.ec2.internal
2026-07-13T21:09:05.740518636Z Collecting performance related data for node ip-10-0-134-97.ec2.internal
2026-07-13T21:09:05.960021500Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-13T21:09:06.296504484Z Gathering data for ns/openshift-kube-scheduler...
2026-07-13T21:09:06.640894665Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-13T21:09:07.019870066Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-13T21:09:07.397098479Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-13T21:09:07.691484633Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-13T21:09:07.865628267Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-13T21:09:08.269736071Z Gathering data for ns/openshift-multus...
2026-07-13T21:09:09.623440694Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-13T21:09:10.905790857Z Gathering data for ns/openshift-host-network...
2026-07-13T21:09:11.211804795Z Gathering data for ns/openshift-network-diagnostics...
2026-07-13T21:09:11.725699620Z Gathering data for ns/openshift-network-node-identity...
2026-07-13T21:09:12.145191299Z Gathering data for ns/openshift-network-operator...
2026-07-13T21:09:12.520353603Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-13T21:09:12.824162846Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-13T21:09:12.870661284Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-13T21:09:13.273256327Z Gathering data for ns/openshift-apiserver-operator...
2026-07-13T21:09:13.556591131Z Gathering data for ns/openshift-apiserver...
2026-07-13T21:09:13.956375984Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-13T21:09:13.996932870Z Collecting kubelet logs for node ip-10-0-128-233.ec2.internal
2026-07-13T21:09:13.999647166Z Collecting kubelet logs for node ip-10-0-128-52.ec2.internal
2026-07-13T21:09:14.002019085Z Collecting kubelet logs for node ip-10-0-134-97.ec2.internal
2026-07-13T21:09:14.274925262Z Gathering data for ns/openshift-controller-manager...
2026-07-13T21:09:14.388529338Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-xlsxm namespace
2026-07-13T21:09:14.392827508Z INFO: Node performance data collection complete.
2026-07-13T21:09:14.653770107Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-13T21:09:15.300918191Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-13T21:09:15.579555431Z Gathering data for ns/openshift-service-ca-operator...
2026-07-13T21:09:15.883865467Z Gathering data for ns/openshift-service-ca...
2026-07-13T21:09:16.220742347Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-13T21:09:17.049677622Z Wrote inspect data to must-gather.
2026-07-13T21:09:17.049725125Z error: inspection completed with the errors occurred while gathering data:
2026-07-13T21:09:17.049725125Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-13T21:09:17.179289120Z error: the server doesn't have a resource type "clusters"
2026-07-13T21:09:17.296288725Z Caches written to disk