2026-05-27T18:18:47.936179308Z [disk usage checker] Started
2026-05-27T18:18:47.939931421Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:18:48.177756332Z Gathering data for ns/openshift-cluster-version...
2026-05-27T18:18:48.347732025Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-27T18:18:48.453947140Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-27T18:18:48.555044846Z Gathering data for ns/default...
2026-05-27T18:18:48.925611963Z Gathering data for ns/openshift...
2026-05-27T18:18:49.419260658Z Gathering data for ns/kube-system...
2026-05-27T18:18:50.144587806Z Gathering data for ns/openshift-etcd...
2026-05-27T18:18:50.497028469Z Waiting on subprocesses to finish execution.
2026-05-27T18:18:50.505482750Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-27T18:18:50.523892833Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-27T18:18:50.532691965Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-27T18:18:50.538894039Z INFO: Gathering HAProxy config files
2026-05-27T18:18:50.541018134Z INFO: Collecting host service logs for crio
2026-05-27T18:18:50.541347758Z INFO: Collecting host service logs for kubelet
2026-05-27T18:18:50.541714451Z INFO: Collecting host service logs for rpm-ostreed
2026-05-27T18:18:50.542091264Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-27T18:18:50.542526023Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-27T18:18:50.542968626Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-27T18:18:50.543297764Z INFO: Collecting host service logs for NetworkManager
2026-05-27T18:18:50.543630265Z INFO: Collecting host service logs for openvswitch
2026-05-27T18:18:50.543945659Z INFO: Collecting host service logs for ovs-configuration
2026-05-27T18:18:50.544277904Z Wrote inspect data to must-gather.
2026-05-27T18:18:50.544296989Z INFO: Collecting host service logs for ovsdb-server
2026-05-27T18:18:50.544372836Z error: inspection completed with the errors occurred while gathering data:
2026-05-27T18:18:50.544372836Z     namespaces "assisted-installer" not found
2026-05-27T18:18:50.544627346Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-27T18:18:50.544993143Z INFO: Waiting for worker host service log collection to complete ...
2026-05-27T18:18:50.561802611Z INFO: Waiting for node performance related collection to complete ...
2026-05-27T18:18:51.142198551Z No resources found in openshift-etcd namespace.
2026-05-27T18:18:51.181036614Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-tslgf
2026-05-27T18:18:51.181088079Z insights-runtime-extractor-bzbwc
2026-05-27T18:18:51.181107351Z insights-runtime-extractor-dwq2j
2026-05-27T18:18:51.181126359Z insights-runtime-extractor-fxn8b
2026-05-27T18:18:51.195578175Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-27T18:18:51.620698932Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-27T18:18:51.702944826Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-27T18:18:51.780814652Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-27T18:18:51.780904021Z INFO: on-disk MachineConfig config collection complete.
2026-05-27T18:18:51.824112406Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-27T18:18:51.887244800Z No resources found
2026-05-27T18:18:51.988963930Z Wrote inspect data to must-gather.
2026-05-27T18:18:52.002153612Z INFO: "metallb-operator" not detected. Skipping.
2026-05-27T18:18:52.131231070Z error: the server doesn't have a resource type "performanceprofile"
2026-05-27T18:18:52.143712020Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-27T18:18:52.143712020Z See 'oc rsync -h' for help and examples
2026-05-27T18:18:52.157426301Z ERROR: No running kube-apiserver pods found
2026-05-27T18:18:52.236007349Z INFO: Worker host service log collection to complete.
2026-05-27T18:18:52.556965891Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-27T18:18:52.570967043Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-27T18:18:52.785686920Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-27T18:18:52.876454401Z INFO: 'previous-logs' folder not found on ip-10-0-129-78.ec2.internal, skipping...
2026-05-27T18:18:52.977655799Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:18:52.991471380Z Wrote inspect data to must-gather.
2026-05-27T18:18:53.207379343Z Wrote inspect data to must-gather.
2026-05-27T18:18:53.226424655Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-27T18:18:53.242607897Z Wrote inspect data to must-gather.
2026-05-27T18:18:53.262796325Z Wrote inspect data to must-gather.
2026-05-27T18:18:53.269088961Z No resources found
2026-05-27T18:18:53.286497690Z Wrote inspect data to must-gather.
2026-05-27T18:18:53.390144207Z error: the server doesn't have a resource type "machineconfigs"
2026-05-27T18:18:53.420857069Z tar: Removing leading `/' from member names
2026-05-27T18:18:53.428093247Z INFO: HAProxy config collection complete.
2026-05-27T18:18:53.571035714Z INFO: 'previous-logs' folder not found on ip-10-0-135-90.ec2.internal, skipping...
2026-05-27T18:18:53.601582694Z Gathering data for ns/kuadrant-system...
2026-05-27T18:18:53.667080106Z INFO: Getting rules from prometheus-k8s-0
2026-05-27T18:18:53.704970802Z Wrote inspect data to must-gather.
2026-05-27T18:18:53.707056510Z Wrote inspect data to must-gather.
2026-05-27T18:18:53.899964517Z INFO: 'previous-logs' folder not found on ip-10-0-141-245.ec2.internal, skipping...
2026-05-27T18:18:53.900009359Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-27T18:18:53.900041165Z INFO: Machine Config Daemon termination log collection complete.
2026-05-27T18:18:54.026987107Z Wrote inspect data to must-gather.
2026-05-27T18:18:54.099103882Z Wrote inspect data to must-gather.
2026-05-27T18:18:54.196491991Z INFO: Getting status/config from prometheus-k8s-0
2026-05-27T18:18:54.257746107Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-27T18:18:54.485915439Z Wrote inspect data to must-gather.
2026-05-27T18:18:54.633506402Z Wrote inspect data to must-gather.
2026-05-27T18:18:54.691788527Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-27T18:18:54.733528466Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-27T18:18:54.925689151Z INFO: INTERCONNECT MODE
2026-05-27T18:18:54.925725295Z INFO: Gathering ovn-kubernetes DBs
2026-05-27T18:18:54.936296955Z Wrote inspect data to must-gather.
2026-05-27T18:18:55.001759316Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-27T18:18:55.152829019Z INFO: Gathering OVN_Northbound from ovnkube-node-f2lpm...
2026-05-27T18:18:55.153144369Z INFO: Gathering OVN_Northbound from ovnkube-node-pdw22...
2026-05-27T18:18:55.153411633Z INFO: Gathering OVN_Northbound from ovnkube-node-pkcfp...
2026-05-27T18:18:55.153722386Z INFO: Gathering OVN_Southbound from ovnkube-node-f2lpm...
2026-05-27T18:18:55.153989183Z INFO: Gathering OVN_Southbound from ovnkube-node-pdw22...
2026-05-27T18:18:55.154242404Z INFO: Gathering OVN_Southbound from ovnkube-node-pkcfp...
2026-05-27T18:18:55.251611739Z W0527 18:18:55.251545    1208 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-27T18:18:55.290116971Z Wrote inspect data to must-gather.
2026-05-27T18:18:55.325471947Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-27T18:18:55.346955499Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-27T18:18:55.539137882Z tar: Removing leading `/' from member names
2026-05-27T18:18:55.562267629Z tar: Removing leading `/' from member names
2026-05-27T18:18:55.594242861Z tar: Removing leading `/' from member names
2026-05-27T18:18:55.656817370Z tar: Removing leading `/' from member names
2026-05-27T18:18:55.692439595Z tar: Removing leading `/' from member names
2026-05-27T18:18:55.699585511Z tar: Removing leading `/' from member names
2026-05-27T18:18:55.781845113Z Gathering data for ns/openshift-monitoring...
2026-05-27T18:18:55.815186374Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-27T18:18:55.916849105Z Wrote inspect data to must-gather.
2026-05-27T18:18:56.240405829Z INFO: Waiting for network log collection to complete ...
2026-05-27T18:18:56.240649714Z INFO: Waiting for ovnk database copies to complete ...
2026-05-27T18:18:56.241188408Z INFO: Copying ovnk databases complete.
2026-05-27T18:18:56.244770259Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-05-27T18:18:56.250363736Z ovnk_database_store/
2026-05-27T18:18:56.250363736Z ovnk_database_store/ovnkube-node-f2lpm_sbdb
2026-05-27T18:18:56.277651001Z ovnk_database_store/ovnkube-node-f2lpm_nbdb
2026-05-27T18:18:56.284379388Z INFO: Getting status from alertmanager-main-0
2026-05-27T18:18:56.286581851Z ovnk_database_store/ovnkube-node-pdw22_nbdb
2026-05-27T18:18:56.298454064Z ovnk_database_store/ovnkube-node-pkcfp_nbdb
2026-05-27T18:18:56.308329592Z ovnk_database_store/ovnkube-node-pdw22_sbdb
2026-05-27T18:18:56.334174865Z ovnk_database_store/ovnkube-node-pkcfp_sbdb
2026-05-27T18:18:56.347815274Z Wrote inspect data to must-gather.
2026-05-27T18:18:56.363347058Z INFO: Network log collection complete.
2026-05-27T18:18:56.653520469Z Wrote inspect data to must-gather.
2026-05-27T18:18:56.953311179Z Wrote inspect data to must-gather.
2026-05-27T18:18:57.271751196Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-27T18:18:57.273804914Z INFO: Fallback to identify the container image from release info
2026-05-27T18:18:57.855010293Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-27T18:18:57.922098594Z Gathering data for ns/openshift-network-console...
2026-05-27T18:18:57.993331379Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:18:57.997268304Z daemonset.apps/perf-node-gather-daemonset created
2026-05-27T18:18:58.157696625Z Waiting for performance profile collector pods to become ready: 1
2026-05-27T18:18:58.488215903Z Gathering data for ns/openshift-console-operator...
2026-05-27T18:18:58.958423987Z Gathering data for ns/openshift-console...
2026-05-27T18:18:59.281457016Z Waiting for performance profile collector pods to become ready: 2
2026-05-27T18:18:59.549294894Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-27T18:19:00.007681365Z Gathering data for ns/openshift-dns-operator...
2026-05-27T18:19:00.149897360Z Wrote inspect data to must-gather.
2026-05-27T18:19:00.354520942Z Gathering data for ns/openshift-dns...
2026-05-27T18:19:00.404027762Z Waiting for performance profile collector pods to become ready: 3
2026-05-27T18:19:00.966443712Z Gathering data for ns/openshift-image-registry...
2026-05-27T18:19:01.456873887Z Gathering data for ns/openshift-ingress-operator...
2026-05-27T18:19:01.537125062Z Waiting for performance profile collector pods to become ready: 4
2026-05-27T18:19:01.836728343Z Gathering data for ns/openshift-ingress...
2026-05-27T18:19:02.674027390Z Gathering data for ns/openshift-ingress-canary...
2026-05-27T18:19:02.677274329Z Waiting for performance profile collector pods to become ready: 5
2026-05-27T18:19:03.002439289Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:19:03.171462729Z Gathering data for ns/openshift-insights...
2026-05-27T18:19:03.807629280Z Waiting for performance profile collector pods to become ready: 6
2026-05-27T18:19:04.920174552Z Waiting for performance profile collector pods to become ready: 7
2026-05-27T18:19:05.190475673Z Gathering data for ns/opendatahub...
2026-05-27T18:19:06.036009504Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-27T18:19:06.304709096Z Collecting performance related data for node ip-10-0-135-90.ec2.internal
2026-05-27T18:19:06.311375910Z Collecting performance related data for node ip-10-0-141-245.ec2.internal
2026-05-27T18:19:06.318117268Z Collecting performance related data for node ip-10-0-129-78.ec2.internal
2026-05-27T18:19:06.762528832Z Gathering data for ns/openshift-lws-operator...
2026-05-27T18:19:08.008934649Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:19:08.228458527Z Gathering data for ns/openshift-config...
2026-05-27T18:19:08.530639184Z Gathering data for ns/openshift-config-managed...
2026-05-27T18:19:09.114275011Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-27T18:19:09.474569946Z Gathering data for ns/openshift-kube-apiserver...
2026-05-27T18:19:09.807852132Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-27T18:19:10.168775411Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-27T18:19:10.532186133Z Gathering data for ns/openshift-kube-scheduler...
2026-05-27T18:19:10.849072724Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-27T18:19:11.219994653Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-27T18:19:11.611766542Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-27T18:19:12.022826375Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-27T18:19:12.640169803Z Gathering data for ns/openshift-multus...
2026-05-27T18:19:13.013776642Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:19:14.041577929Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-27T18:19:14.945223053Z Collecting kubelet logs for node ip-10-0-135-90.ec2.internal
2026-05-27T18:19:14.947432627Z Collecting kubelet logs for node ip-10-0-141-245.ec2.internal
2026-05-27T18:19:14.951766413Z Collecting kubelet logs for node ip-10-0-129-78.ec2.internal
2026-05-27T18:19:15.320228545Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-27T18:19:15.325776507Z INFO: Node performance data collection complete.
2026-05-27T18:19:15.691788563Z Gathering data for ns/openshift-host-network...
2026-05-27T18:19:16.043100935Z Gathering data for ns/openshift-network-diagnostics...
2026-05-27T18:19:16.629205748Z Gathering data for ns/openshift-network-node-identity...
2026-05-27T18:19:17.093525212Z Gathering data for ns/openshift-network-operator...
2026-05-27T18:19:17.533374887Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-27T18:19:17.859783777Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-27T18:19:18.018430451Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:19:18.366321049Z Gathering data for ns/openshift-apiserver-operator...
2026-05-27T18:19:18.722454201Z Gathering data for ns/openshift-apiserver...
2026-05-27T18:19:19.212335415Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-27T18:19:19.538146612Z Gathering data for ns/openshift-controller-manager...
2026-05-27T18:19:19.884830676Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-27T18:19:20.590512160Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-27T18:19:20.943458034Z Gathering data for ns/openshift-service-ca-operator...
2026-05-27T18:19:21.288792241Z Gathering data for ns/openshift-service-ca...
2026-05-27T18:19:21.734336999Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-27T18:19:22.746367675Z Wrote inspect data to must-gather.
2026-05-27T18:19:22.746408805Z error: inspection completed with the errors occurred while gathering data:
2026-05-27T18:19:22.746408805Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-27T18:19:22.883491012Z error: the server doesn't have a resource type "clusters"
2026-05-27T18:19:23.022993566Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-27T18:19:23.071262991Z Caches written to disk