2026-04-23T17:05:31.284985269Z [disk usage checker] Started
2026-04-23T17:05:31.319453788Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-23T17:05:31.617006466Z Gathering data for ns/openshift-cluster-version...
2026-04-23T17:05:31.761254800Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T17:05:31.857527386Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T17:05:31.953674647Z Gathering data for ns/default...
2026-04-23T17:05:32.292998143Z Gathering data for ns/openshift...
2026-04-23T17:05:32.712384613Z Gathering data for ns/kube-system...
2026-04-23T17:05:33.349765013Z Gathering data for ns/openshift-etcd...
2026-04-23T17:05:33.667159723Z Wrote inspect data to must-gather.
2026-04-23T17:05:33.667193852Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T17:05:33.667193852Z     namespaces "assisted-installer" not found
2026-04-23T17:05:33.998524296Z Waiting on subprocesses to finish execution.
2026-04-23T17:05:34.005141717Z INFO: Gathering HAProxy config files
2026-04-23T17:05:34.007086028Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-23T17:05:34.007957614Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-23T17:05:34.012868687Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-23T17:05:34.016535695Z INFO: Collecting host service logs for crio
2026-04-23T17:05:34.016902129Z INFO: Collecting host service logs for kubelet
2026-04-23T17:05:34.017267285Z INFO: Collecting host service logs for rpm-ostreed
2026-04-23T17:05:34.017643779Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-23T17:05:34.018023939Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-23T17:05:34.018406349Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-23T17:05:34.018762217Z INFO: Collecting host service logs for NetworkManager
2026-04-23T17:05:34.019150981Z INFO: Collecting host service logs for openvswitch
2026-04-23T17:05:34.019529255Z INFO: Collecting host service logs for ovs-configuration
2026-04-23T17:05:34.019896434Z INFO: Collecting host service logs for ovsdb-server
2026-04-23T17:05:34.020251744Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-23T17:05:34.020703552Z INFO: Waiting for worker host service log collection to complete ...
2026-04-23T17:05:34.039751520Z INFO: Waiting for node performance related collection to complete ...
2026-04-23T17:05:35.087593610Z No resources found in openshift-etcd namespace.
2026-04-23T17:05:35.315395713Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-23T17:05:35.350102147Z error: the server doesn't have a resource type "performanceprofile"
2026-04-23T17:05:35.396051552Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-d97rf
2026-04-23T17:05:35.396051552Z insights-runtime-extractor-g4cmx
2026-04-23T17:05:35.396051552Z insights-runtime-extractor-vrfds
2026-04-23T17:05:35.396051552Z insights-runtime-extractor-z2gnb
2026-04-23T17:05:35.411268609Z No resources found
2026-04-23T17:05:35.474389042Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-23T17:05:35.474467991Z INFO: on-disk MachineConfig config collection complete.
2026-04-23T17:05:35.541720519Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-23T17:05:35.664732485Z INFO: Worker host service log collection to complete.
2026-04-23T17:05:35.763936046Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-23T17:05:35.789106034Z ERROR: No running kube-apiserver pods found
2026-04-23T17:05:35.843647052Z Wrote inspect data to must-gather.
2026-04-23T17:05:35.888970059Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-23T17:05:35.985880870Z Gathering data for ns/kuadrant-system...
2026-04-23T17:05:36.124370675Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-23T17:05:36.192267806Z INFO: "metallb-operator" not detected. Skipping.
2026-04-23T17:05:36.254216478Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T17:05:36.311719151Z No resources found
2026-04-23T17:05:36.498249140Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-23T17:05:36.589877610Z Wrote inspect data to must-gather.
2026-04-23T17:05:36.641739212Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-23T17:05:36.683123742Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-23T17:05:36.724763911Z Wrote inspect data to must-gather.
2026-04-23T17:05:36.725461459Z INFO: 'previous-logs' folder not found on ip-10-0-128-198.ec2.internal, skipping...
2026-04-23T17:05:36.731061772Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-23T17:05:36.731061772Z See 'oc rsync -h' for help and examples
2026-04-23T17:05:36.768428102Z Wrote inspect data to must-gather.
2026-04-23T17:05:36.874694297Z Wrote inspect data to must-gather.
2026-04-23T17:05:36.935450220Z Wrote inspect data to must-gather.
2026-04-23T17:05:36.978554894Z error: the server doesn't have a resource type "machineconfigs"
2026-04-23T17:05:36.984116840Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T17:05:37.117111913Z tar: Removing leading `/' from member names
2026-04-23T17:05:37.126073015Z INFO: HAProxy config collection complete.
2026-04-23T17:05:37.193867606Z INFO: 'previous-logs' folder not found on ip-10-0-136-190.ec2.internal, skipping...
2026-04-23T17:05:37.253727140Z INFO: Getting rules from prometheus-k8s-0
2026-04-23T17:05:37.285020982Z Wrote inspect data to must-gather.
2026-04-23T17:05:37.331886651Z W0423 17:05:37.331837     997 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-23T17:05:37.355762447Z Wrote inspect data to must-gather.
2026-04-23T17:05:37.367968991Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T17:05:37.622788992Z Gathering data for ns/openshift-monitoring...
2026-04-23T17:05:37.633783869Z INFO: 'previous-logs' folder not found on ip-10-0-136-27.ec2.internal, skipping...
2026-04-23T17:05:37.633895698Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-23T17:05:37.633945898Z INFO: Machine Config Daemon termination log collection complete.
2026-04-23T17:05:37.773230968Z Wrote inspect data to must-gather.
2026-04-23T17:05:37.775383821Z Wrote inspect data to must-gather.
2026-04-23T17:05:37.794780062Z INFO: Getting status/config from prometheus-k8s-0
2026-04-23T17:05:37.975076354Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-23T17:05:38.112383520Z Wrote inspect data to must-gather.
2026-04-23T17:05:38.294795200Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-23T17:05:38.309012507Z Wrote inspect data to must-gather.
2026-04-23T17:05:38.504329414Z INFO: INTERCONNECT MODE
2026-04-23T17:05:38.504329414Z INFO: Gathering ovn-kubernetes DBs
2026-04-23T17:05:38.523709519Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-23T17:05:38.623574577Z Wrote inspect data to must-gather.
2026-04-23T17:05:38.708547173Z INFO: Gathering OVN_Northbound from ovnkube-node-jbv88...
2026-04-23T17:05:38.708799216Z INFO: Gathering OVN_Northbound from ovnkube-node-nfkqz...
2026-04-23T17:05:38.709028620Z INFO: Gathering OVN_Northbound from ovnkube-node-q6qml...
2026-04-23T17:05:38.709391479Z INFO: Gathering OVN_Southbound from ovnkube-node-jbv88...
2026-04-23T17:05:38.709558990Z INFO: Gathering OVN_Southbound from ovnkube-node-nfkqz...
2026-04-23T17:05:38.709830050Z INFO: Gathering OVN_Southbound from ovnkube-node-q6qml...
2026-04-23T17:05:38.801418948Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-23T17:05:39.016413383Z tar: Removing leading `/' from member names
2026-04-23T17:05:39.038611648Z tar: Removing leading `/' from member names
2026-04-23T17:05:39.106452207Z tar: Removing leading `/' from member names
2026-04-23T17:05:39.116093190Z tar: Removing leading `/' from member names
2026-04-23T17:05:39.166401881Z Wrote inspect data to must-gather.
2026-04-23T17:05:39.220552138Z tar: Removing leading `/' from member names
2026-04-23T17:05:39.267698190Z tar: Removing leading `/' from member names
2026-04-23T17:05:39.356046689Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-23T17:05:39.564681237Z Wrote inspect data to must-gather.
2026-04-23T17:05:39.881332631Z W0423 17:05:39.881262    1388 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-23T17:05:39.891035461Z Wrote inspect data to must-gather.
2026-04-23T17:05:39.897877875Z INFO: Getting status from alertmanager-main-0
2026-04-23T17:05:39.907094061Z INFO: Waiting for network log collection to complete ...
2026-04-23T17:05:39.907161684Z INFO: Waiting for ovnk database copies to complete ...
2026-04-23T17:05:39.907743623Z INFO: Copying ovnk databases complete.
2026-04-23T17:05:39.909625295Z 4.9M	must-gather/network_logs/ovnk_database_store
2026-04-23T17:05:39.912469800Z ovnk_database_store/
2026-04-23T17:05:39.912469800Z ovnk_database_store/ovnkube-node-jbv88_sbdb
2026-04-23T17:05:39.945165577Z ovnk_database_store/ovnkube-node-q6qml_sbdb
2026-04-23T17:05:39.973795064Z ovnk_database_store/ovnkube-node-nfkqz_sbdb
2026-04-23T17:05:40.005738094Z ovnk_database_store/ovnkube-node-jbv88_nbdb
2026-04-23T17:05:40.020429147Z ovnk_database_store/ovnkube-node-nfkqz_nbdb
2026-04-23T17:05:40.036734509Z ovnk_database_store/ovnkube-node-q6qml_nbdb
2026-04-23T17:05:40.054646333Z INFO: Network log collection complete.
2026-04-23T17:05:40.263517965Z Gathering data for ns/openshift-network-console...
2026-04-23T17:05:40.311045190Z Wrote inspect data to must-gather.
2026-04-23T17:05:40.692382105Z Wrote inspect data to must-gather.
2026-04-23T17:05:40.824538295Z Gathering data for ns/openshift-console-operator...
2026-04-23T17:05:41.018378050Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-23T17:05:41.022256739Z INFO: Fallback to identify the container image from release info
2026-04-23T17:05:41.310117625Z Gathering data for ns/openshift-console...
2026-04-23T17:05:41.533238943Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-23T17:05:41.534046399Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-23T17:05:41.682416397Z daemonset.apps/perf-node-gather-daemonset created
2026-04-23T17:05:41.838852740Z Wrote inspect data to must-gather.
2026-04-23T17:05:41.846279540Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-23T17:05:41.848455305Z Waiting for performance profile collector pods to become ready: 1
2026-04-23T17:05:42.234639463Z Gathering data for ns/openshift-dns-operator...
2026-04-23T17:05:42.535283791Z Gathering data for ns/openshift-dns...
2026-04-23T17:05:42.984849847Z Waiting for performance profile collector pods to become ready: 2
2026-04-23T17:05:43.103874661Z Gathering data for ns/openshift-image-registry...
2026-04-23T17:05:43.546278753Z Gathering data for ns/openshift-ingress-operator...
2026-04-23T17:05:43.908325782Z Gathering data for ns/openshift-ingress...
2026-04-23T17:05:44.115446834Z Waiting for performance profile collector pods to become ready: 3
2026-04-23T17:05:44.452872959Z Gathering data for ns/openshift-ingress-canary...
2026-04-23T17:05:44.869894942Z Gathering data for ns/openshift-insights...
2026-04-23T17:05:45.244095250Z Waiting for performance profile collector pods to become ready: 4
2026-04-23T17:05:46.370682123Z Waiting for performance profile collector pods to become ready: 5
2026-04-23T17:05:46.550717402Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-23T17:05:47.495941892Z Waiting for performance profile collector pods to become ready: 6
2026-04-23T17:05:47.572685699Z Gathering data for ns/openshift-lws-operator...
2026-04-23T17:05:48.094514495Z Gathering data for ns/kserve...
2026-04-23T17:05:48.640536983Z Waiting for performance profile collector pods to become ready: 7
2026-04-23T17:05:49.762648637Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-23T17:05:49.865958989Z Gathering data for ns/openshift-config...
2026-04-23T17:05:50.098234893Z Collecting performance related data for node ip-10-0-128-198.ec2.internal
2026-04-23T17:05:50.107146249Z Collecting performance related data for node ip-10-0-136-190.ec2.internal
2026-04-23T17:05:50.121683225Z Collecting performance related data for node ip-10-0-136-27.ec2.internal
2026-04-23T17:05:50.215455403Z Gathering data for ns/openshift-config-managed...
2026-04-23T17:05:50.751729615Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-23T17:05:51.139025115Z Gathering data for ns/openshift-kube-apiserver...
2026-04-23T17:05:51.540264935Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-23T17:05:51.559419253Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-23T17:05:51.919011893Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-23T17:05:52.222314399Z Gathering data for ns/openshift-kube-scheduler...
2026-04-23T17:05:52.512144005Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-23T17:05:52.852442177Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-23T17:05:53.243119261Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-23T17:05:53.571608800Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-23T17:05:54.128103371Z Gathering data for ns/openshift-multus...
2026-04-23T17:05:55.481553702Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-23T17:05:56.569368178Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-23T17:05:57.168461658Z Gathering data for ns/openshift-host-network...
2026-04-23T17:05:57.503716575Z Gathering data for ns/openshift-network-diagnostics...
2026-04-23T17:05:58.073599229Z Gathering data for ns/openshift-network-node-identity...
2026-04-23T17:05:58.515953560Z Gathering data for ns/openshift-network-operator...
2026-04-23T17:05:58.910326986Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-23T17:05:59.001707776Z Collecting kubelet logs for node ip-10-0-128-198.ec2.internal
2026-04-23T17:05:59.003448496Z Collecting kubelet logs for node ip-10-0-136-190.ec2.internal
2026-04-23T17:05:59.005148249Z Collecting kubelet logs for node ip-10-0-136-27.ec2.internal
2026-04-23T17:05:59.270225312Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-23T17:05:59.444486323Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-23T17:05:59.449974602Z INFO: Node performance data collection complete.
2026-04-23T17:05:59.806460514Z Gathering data for ns/openshift-apiserver-operator...
2026-04-23T17:06:00.129525375Z Gathering data for ns/openshift-apiserver...
2026-04-23T17:06:00.577174335Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-23T17:06:00.846843504Z Gathering data for ns/openshift-controller-manager...
2026-04-23T17:06:01.173749174Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-23T17:06:01.574210969Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-23T17:06:01.857883426Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-23T17:06:02.144208038Z Gathering data for ns/openshift-service-ca-operator...
2026-04-23T17:06:02.455767632Z Gathering data for ns/openshift-service-ca...
2026-04-23T17:06:02.813872317Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-23T17:06:03.719103493Z Wrote inspect data to must-gather.
2026-04-23T17:06:03.719155954Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T17:06:03.719155954Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-23T17:06:03.860083569Z error: the server doesn't have a resource type "clusters"
2026-04-23T17:06:04.031246242Z Caches written to disk