2026-07-08T07:18:58.716713572Z [disk usage checker] Started
2026-07-08T07:18:58.720550142Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-08T07:18:58.926106979Z Gathering data for ns/openshift-cluster-version...
2026-07-08T07:18:59.056927766Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-08T07:18:59.146354937Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T07:18:59.229145005Z Gathering data for ns/default...
2026-07-08T07:18:59.546270634Z Gathering data for ns/openshift...
2026-07-08T07:18:59.957076118Z Gathering data for ns/kube-system...
2026-07-08T07:19:00.703755456Z Gathering data for ns/openshift-etcd...
2026-07-08T07:19:01.033485266Z Wrote inspect data to must-gather.
2026-07-08T07:19:01.033520072Z error: inspection completed with the errors occurred while gathering data:
2026-07-08T07:19:01.033520072Z     namespaces "assisted-installer" not found
2026-07-08T07:19:01.541796470Z Waiting on subprocesses to finish execution.
2026-07-08T07:19:01.550748154Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-08T07:19:01.550844839Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-08T07:19:01.556379788Z INFO: Gathering HAProxy config files
2026-07-08T07:19:01.561386138Z INFO: Collecting host service logs for crio
2026-07-08T07:19:01.561724864Z INFO: Collecting host service logs for kubelet
2026-07-08T07:19:01.562137638Z INFO: Collecting host service logs for rpm-ostreed
2026-07-08T07:19:01.562481323Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-08T07:19:01.563808386Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-08T07:19:01.564131006Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-08T07:19:01.564453244Z INFO: Collecting host service logs for NetworkManager
2026-07-08T07:19:01.564493240Z Executing Istio gather script
2026-07-08T07:19:01.564789173Z INFO: Collecting host service logs for openvswitch
2026-07-08T07:19:01.565175521Z INFO: Collecting host service logs for ovs-configuration
2026-07-08T07:19:01.565556925Z INFO: Collecting host service logs for ovsdb-server
2026-07-08T07:19:01.565904445Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-08T07:19:01.566285274Z INFO: Waiting for worker host service log collection to complete ...
2026-07-08T07:19:01.573874966Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-08T07:19:01.601342900Z INFO: Waiting for node performance related collection to complete ...
2026-07-08T07:19:02.257530366Z No resources found in openshift-etcd namespace.
2026-07-08T07:19:02.384884366Z Wrote inspect data to must-gather.
2026-07-08T07:19:02.445526992Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-08T07:19:02.445558637Z INFO: on-disk MachineConfig config collection complete.
2026-07-08T07:19:02.713824562Z INFO: "metallb-operator" not detected. Skipping.
2026-07-08T07:19:02.765934064Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-08T07:19:02.866839459Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-08T07:19:02.982567014Z error: the server doesn't have a resource type "performanceprofile"
2026-07-08T07:19:02.993990358Z No resources found
2026-07-08T07:19:03.034852515Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-hjqbx
2026-07-08T07:19:03.034903801Z insights-runtime-extractor-nnhgx
2026-07-08T07:19:03.034924805Z insights-runtime-extractor-swt9c
2026-07-08T07:19:03.034946634Z insights-runtime-extractor-xmcn2
2026-07-08T07:19:03.038661384Z INFO: Worker host service log collection to complete.
2026-07-08T07:19:03.092454284Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-08T07:19:03.232273337Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-08T07:19:03.319448095Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-08T07:19:03.868502438Z Wrote inspect data to must-gather.
2026-07-08T07:19:03.891233954Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-08T07:19:03.904332661Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-08T07:19:03.924126631Z No resources found
2026-07-08T07:19:03.975079504Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-08T07:19:04.061247626Z Wrote inspect data to must-gather.
2026-07-08T07:19:04.102307027Z Wrote inspect data to must-gather.
2026-07-08T07:19:04.109342322Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-08T07:19:04.109342322Z See 'oc rsync -h' for help and examples
2026-07-08T07:19:04.135695733Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T07:19:04.191287141Z Wrote inspect data to must-gather.
2026-07-08T07:19:04.206504383Z INFO: 'previous-logs' folder not found on ip-10-0-136-197.ec2.internal, skipping...
2026-07-08T07:19:04.217838622Z ERROR: No running kube-apiserver pods found
2026-07-08T07:19:04.330724505Z Wrote inspect data to must-gather.
2026-07-08T07:19:04.338012542Z tar: Removing leading `/' from member names
2026-07-08T07:19:04.342622178Z INFO: HAProxy config collection complete.
2026-07-08T07:19:04.419215325Z Wrote inspect data to must-gather.
2026-07-08T07:19:04.532009894Z INFO: 'previous-logs' folder not found on ip-10-0-137-201.ec2.internal, skipping...
2026-07-08T07:19:04.553746808Z Gathering data for ns/openshift-monitoring...
2026-07-08T07:19:04.588962865Z error: the server doesn't have a resource type "machineconfigs"
2026-07-08T07:19:04.692140004Z INFO: Getting rules from prometheus-k8s-0
2026-07-08T07:19:04.763706894Z Wrote inspect data to must-gather.
2026-07-08T07:19:04.809052173Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-08T07:19:04.854650406Z INFO: 'previous-logs' folder not found on ip-10-0-140-161.ec2.internal, skipping...
2026-07-08T07:19:04.854756949Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-08T07:19:04.854806277Z INFO: Machine Config Daemon termination log collection complete.
2026-07-08T07:19:05.071815604Z Wrote inspect data to must-gather.
2026-07-08T07:19:05.085880048Z Wrote inspect data to must-gather.
2026-07-08T07:19:05.183218447Z INFO: Getting status/config from prometheus-k8s-0
2026-07-08T07:19:05.261145873Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-08T07:19:05.281417871Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-08T07:19:05.358729093Z Wrote inspect data to must-gather.
2026-07-08T07:19:05.532623563Z Wrote inspect data to must-gather.
2026-07-08T07:19:05.634494922Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-08T07:19:05.686914733Z INFO: INTERCONNECT MODE
2026-07-08T07:19:05.686947017Z INFO: Gathering ovn-kubernetes DBs
2026-07-08T07:19:05.777943071Z Wrote inspect data to must-gather.
2026-07-08T07:19:05.856093267Z INFO: Gathering OVN_Northbound from ovnkube-node-48xw7...
2026-07-08T07:19:05.856410867Z INFO: Gathering OVN_Northbound from ovnkube-node-ctjb4...
2026-07-08T07:19:05.856691378Z INFO: Gathering OVN_Northbound from ovnkube-node-kc6bb...
2026-07-08T07:19:05.856962097Z INFO: Gathering OVN_Southbound from ovnkube-node-48xw7...
2026-07-08T07:19:05.857239710Z INFO: Gathering OVN_Southbound from ovnkube-node-ctjb4...
2026-07-08T07:19:05.857519161Z INFO: Gathering OVN_Southbound from ovnkube-node-kc6bb...
2026-07-08T07:19:05.904443287Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-08T07:19:06.161761166Z tar: Removing leading `/' from member names
2026-07-08T07:19:06.170989459Z Wrote inspect data to must-gather.
2026-07-08T07:19:06.227685850Z tar: Removing leading `/' from member names
2026-07-08T07:19:06.229383984Z tar: Removing leading `/' from member names
2026-07-08T07:19:06.239711374Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-08T07:19:06.271968408Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-08T07:19:06.286909262Z tar: Removing leading `/' from member names
2026-07-08T07:19:06.302812679Z tar: Removing leading `/' from member names
2026-07-08T07:19:06.308514387Z tar: Removing leading `/' from member names
2026-07-08T07:19:06.342976234Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-08T07:19:06.472244136Z Wrote inspect data to must-gather.
2026-07-08T07:19:06.502738741Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:06.630410232Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-08T07:19:06.768443424Z Wrote inspect data to must-gather.
2026-07-08T07:19:06.802531833Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:06.805153068Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-08T07:19:06.872217485Z INFO: Waiting for network log collection to complete ...
2026-07-08T07:19:06.872253494Z INFO: Waiting for ovnk database copies to complete ...
2026-07-08T07:19:06.872640352Z INFO: Copying ovnk databases complete.
2026-07-08T07:19:06.874191416Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-08T07:19:06.876425956Z ovnk_database_store/
2026-07-08T07:19:06.876425956Z ovnk_database_store/ovnkube-node-kc6bb_nbdb
2026-07-08T07:19:06.889975708Z ovnk_database_store/ovnkube-node-ctjb4_sbdb
2026-07-08T07:19:06.915288168Z ovnk_database_store/ovnkube-node-48xw7_sbdb
2026-07-08T07:19:06.939297874Z ovnk_database_store/ovnkube-node-ctjb4_nbdb
2026-07-08T07:19:06.947192023Z ovnk_database_store/ovnkube-node-48xw7_nbdb
2026-07-08T07:19:06.957286644Z ovnk_database_store/ovnkube-node-kc6bb_sbdb
2026-07-08T07:19:06.971314171Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-08T07:19:06.982303282Z INFO: Network log collection complete.
2026-07-08T07:19:07.072545269Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:07.108121113Z Wrote inspect data to must-gather.
2026-07-08T07:19:07.112641470Z INFO: Getting status from alertmanager-main-0
2026-07-08T07:19:07.176303070Z Gathering data for ns/openshift-network-console...
2026-07-08T07:19:07.304566704Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:07.310338769Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-08T07:19:07.454556367Z Wrote inspect data to must-gather.
2026-07-08T07:19:07.523838163Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:07.657913990Z Gathering data for ns/openshift-console-operator...
2026-07-08T07:19:07.756332247Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:07.759997132Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-08T07:19:07.776187272Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-08T07:19:07.782730788Z INFO: Fallback to identify the container image from release info
2026-07-08T07:19:07.999050829Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:08.072413280Z Gathering data for ns/openshift-console...
2026-07-08T07:19:08.216657285Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:08.219787282Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-08T07:19:08.390077394Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:08.436238975Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-08T07:19:08.597188443Z daemonset.apps/perf-node-gather-daemonset created
2026-07-08T07:19:08.627781865Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-08T07:19:08.639036838Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:08.643403132Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-08T07:19:08.754998497Z Waiting for performance profile collector pods to become ready: 1
2026-07-08T07:19:08.837521206Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:08.906000970Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-07-08T07:19:09.064081278Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:09.067650550Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-08T07:19:09.232735689Z Gathering data for ns/openshift-dns-operator...
2026-07-08T07:19:09.271495223Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:09.470352718Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:09.472908976Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-08T07:19:09.574286185Z Gathering data for ns/openshift-dns...
2026-07-08T07:19:09.645228361Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:09.834326980Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:09.838004460Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-08T07:19:09.912717244Z Waiting for performance profile collector pods to become ready: 2
2026-07-08T07:19:10.022883117Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:10.222409050Z Gathering data for ns/openshift-image-registry...
2026-07-08T07:19:10.234363142Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:10.237692270Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-08T07:19:10.421540318Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:10.612002639Z Wrote inspect data to /must-gather/istio.
2026-07-08T07:19:10.768444166Z Gathering data for ns/openshift-ingress-operator...
2026-07-08T07:19:11.050485770Z Waiting for performance profile collector pods to become ready: 3
2026-07-08T07:19:11.139317565Z Gathering data for ns/openshift-ingress...
2026-07-08T07:19:11.364847118Z Done executing Istio gather script
2026-07-08T07:19:11.477653619Z Gathering data for ns/openshift-ingress-canary...
2026-07-08T07:19:11.845187592Z Gathering data for ns/openshift-insights...
2026-07-08T07:19:12.191618627Z Waiting for performance profile collector pods to become ready: 4
2026-07-08T07:19:13.316812373Z Waiting for performance profile collector pods to become ready: 5
2026-07-08T07:19:13.910877569Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-08T07:19:14.045703629Z Gathering data for ns/kserve...
2026-07-08T07:19:14.442331104Z Waiting for performance profile collector pods to become ready: 6
2026-07-08T07:19:15.574729843Z Waiting for performance profile collector pods to become ready: 7
2026-07-08T07:19:15.637777610Z Gathering data for ns/openshift-config...
2026-07-08T07:19:15.889189079Z Gathering data for ns/openshift-config-managed...
2026-07-08T07:19:16.353364700Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-08T07:19:16.646032772Z Gathering data for ns/openshift-kube-apiserver...
2026-07-08T07:19:16.713388034Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-08T07:19:16.916431435Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-08T07:19:16.989521328Z Collecting performance related data for node ip-10-0-140-161.ec2.internal
2026-07-08T07:19:16.996918309Z Collecting performance related data for node ip-10-0-136-197.ec2.internal
2026-07-08T07:19:17.004543293Z Collecting performance related data for node ip-10-0-137-201.ec2.internal
2026-07-08T07:19:17.258019268Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-08T07:19:17.674448430Z Gathering data for ns/openshift-kube-scheduler...
2026-07-08T07:19:18.016954096Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-08T07:19:18.407651908Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-08T07:19:18.768918083Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-08T07:19:18.915513699Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-08T07:19:19.089182143Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-08T07:19:19.615807957Z Gathering data for ns/openshift-multus...
2026-07-08T07:19:20.986058006Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-08T07:19:22.218974333Z Gathering data for ns/openshift-host-network...
2026-07-08T07:19:22.531703774Z Gathering data for ns/openshift-network-diagnostics...
2026-07-08T07:19:23.116318560Z Gathering data for ns/openshift-network-node-identity...
2026-07-08T07:19:23.550941726Z Gathering data for ns/openshift-network-operator...
2026-07-08T07:19:23.913032556Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-08T07:19:23.920176932Z [disk usage checker] Volume usage percentage: current = 21 ; allowed = 70
2026-07-08T07:19:24.206910923Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-08T07:19:24.709170944Z Gathering data for ns/openshift-apiserver-operator...
2026-07-08T07:19:25.000374463Z Gathering data for ns/openshift-apiserver...
2026-07-08T07:19:25.400509072Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-08T07:19:25.654928573Z Gathering data for ns/openshift-controller-manager...
2026-07-08T07:19:25.745322360Z Collecting kubelet logs for node ip-10-0-140-161.ec2.internal
2026-07-08T07:19:25.747058083Z Collecting kubelet logs for node ip-10-0-136-197.ec2.internal
2026-07-08T07:19:25.748735252Z Collecting kubelet logs for node ip-10-0-137-201.ec2.internal
2026-07-08T07:19:25.972091724Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-08T07:19:26.143916662Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-tqktm namespace
2026-07-08T07:19:26.148530354Z INFO: Node performance data collection complete.
2026-07-08T07:19:26.626880115Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-08T07:19:26.934068653Z Gathering data for ns/openshift-service-ca-operator...
2026-07-08T07:19:27.221379323Z Gathering data for ns/openshift-service-ca...
2026-07-08T07:19:27.548879768Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-08T07:19:28.421673998Z Wrote inspect data to must-gather.
2026-07-08T07:19:28.421720445Z error: inspection completed with the errors occurred while gathering data:
2026-07-08T07:19:28.421720445Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-08T07:19:28.553401478Z error: the server doesn't have a resource type "clusters"
2026-07-08T07:19:28.701516830Z Caches written to disk