2026-04-20T07:40:50.360218889Z [disk usage checker] Started
2026-04-20T07:40:50.364214969Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T07:40:50.620051515Z Gathering data for ns/openshift-cluster-version...
2026-04-20T07:40:50.774292102Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T07:40:50.868722906Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T07:40:51.042472357Z Gathering data for ns/default...
2026-04-20T07:40:51.384690484Z Gathering data for ns/openshift...
2026-04-20T07:40:51.839820874Z Gathering data for ns/kube-system...
2026-04-20T07:40:52.586879573Z Gathering data for ns/openshift-etcd...
2026-04-20T07:40:52.918676071Z Waiting on subprocesses to finish execution.
2026-04-20T07:40:52.924671015Z INFO: Gathering HAProxy config files
2026-04-20T07:40:52.926060460Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-20T07:40:52.933938976Z INFO: Collecting host service logs for crio
2026-04-20T07:40:52.934251426Z INFO: Collecting host service logs for kubelet
2026-04-20T07:40:52.936633602Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-20T07:40:52.937046006Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-20T07:40:52.937492400Z INFO: Collecting host service logs for rpm-ostreed
2026-04-20T07:40:52.937768184Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-20T07:40:52.938047440Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-20T07:40:52.938357202Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-20T07:40:52.939487876Z INFO: Collecting host service logs for NetworkManager
2026-04-20T07:40:52.939804924Z INFO: Collecting host service logs for openvswitch
2026-04-20T07:40:52.940110397Z INFO: Collecting host service logs for ovs-configuration
2026-04-20T07:40:52.940446215Z INFO: Collecting host service logs for ovsdb-server
2026-04-20T07:40:52.940781511Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-20T07:40:52.941102657Z INFO: Waiting for worker host service log collection to complete ...
2026-04-20T07:40:52.994539640Z Wrote inspect data to must-gather.
2026-04-20T07:40:52.994579417Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T07:40:52.994579417Z     namespaces "assisted-installer" not found
2026-04-20T07:40:52.997912301Z INFO: Waiting for node performance related collection to complete ...
2026-04-20T07:40:53.345812412Z No resources found in openshift-etcd namespace.
2026-04-20T07:40:53.409771548Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-20T07:40:53.587157391Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-20T07:40:53.616985292Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-20T07:40:54.076300827Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-jlbv6
2026-04-20T07:40:54.076380183Z insights-runtime-extractor-fd69r
2026-04-20T07:40:54.076407827Z insights-runtime-extractor-fr8mx
2026-04-20T07:40:54.076430766Z insights-runtime-extractor-fscsr
2026-04-20T07:40:54.116660778Z Wrote inspect data to must-gather.
2026-04-20T07:40:54.134363872Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-20T07:40:54.134363872Z INFO: on-disk MachineConfig config collection complete.
2026-04-20T07:40:54.144805133Z error: the server doesn't have a resource type "performanceprofile"
2026-04-20T07:40:54.160568007Z No resources found
2026-04-20T07:40:54.179150602Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-20T07:40:54.384597968Z INFO: "metallb-operator" not detected. Skipping.
2026-04-20T07:40:54.702129236Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-20T07:40:54.703191060Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-20T07:40:54.789731554Z INFO: Worker host service log collection to complete.
2026-04-20T07:40:55.019175048Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-20T07:40:55.019175048Z See 'oc rsync -h' for help and examples
2026-04-20T07:40:55.114364983Z Wrote inspect data to must-gather.
2026-04-20T07:40:55.359201834Z Wrote inspect data to must-gather.
2026-04-20T07:40:55.367563286Z Gathering data for ns/kuadrant-system...
2026-04-20T07:40:55.393312021Z Wrote inspect data to must-gather.
2026-04-20T07:40:55.425188079Z No resources found
2026-04-20T07:40:55.445413460Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T07:40:55.463903012Z INFO: HAProxy config collection complete.
2026-04-20T07:40:55.544609421Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-20T07:40:55.606215168Z Wrote inspect data to must-gather.
2026-04-20T07:40:55.606512225Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T07:40:55.679433980Z ERROR: No running kube-apiserver pods found
2026-04-20T07:40:55.783627775Z Wrote inspect data to must-gather.
2026-04-20T07:40:55.855728584Z INFO: Getting rules from prometheus-k8s-0
2026-04-20T07:40:55.868348115Z INFO: 'previous-logs' folder not found on ip-10-0-130-105.ec2.internal, skipping...
2026-04-20T07:40:55.964960843Z error: the server doesn't have a resource type "machineconfigs"
2026-04-20T07:40:56.021996660Z Wrote inspect data to must-gather.
2026-04-20T07:40:56.273831888Z INFO: 'previous-logs' folder not found on ip-10-0-138-178.ec2.internal, skipping...
2026-04-20T07:40:56.320046011Z Wrote inspect data to must-gather.
2026-04-20T07:40:56.441104014Z Wrote inspect data to must-gather.
2026-04-20T07:40:56.491186118Z INFO: Getting status/config from prometheus-k8s-0
2026-04-20T07:40:56.669696809Z Wrote inspect data to must-gather.
2026-04-20T07:40:56.742002900Z INFO: 'previous-logs' folder not found on ip-10-0-142-100.ec2.internal, skipping...
2026-04-20T07:40:56.742048648Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-20T07:40:56.742059610Z INFO: Machine Config Daemon termination log collection complete.
2026-04-20T07:40:56.802861373Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T07:40:56.830497157Z Wrote inspect data to must-gather.
2026-04-20T07:40:56.911276074Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-20T07:40:57.052432932Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-20T07:40:57.200725670Z INFO: INTERCONNECT MODE
2026-04-20T07:40:57.200759834Z INFO: Gathering ovn-kubernetes DBs
2026-04-20T07:40:57.259969885Z Wrote inspect data to must-gather.
2026-04-20T07:40:57.342691786Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-20T07:40:57.439687456Z INFO: Gathering OVN_Northbound from ovnkube-node-5bfnd...
2026-04-20T07:40:57.439940196Z INFO: Gathering OVN_Northbound from ovnkube-node-qvqrn...
2026-04-20T07:40:57.440174455Z INFO: Gathering OVN_Northbound from ovnkube-node-vz4hl...
2026-04-20T07:40:57.440454577Z INFO: Gathering OVN_Southbound from ovnkube-node-5bfnd...
2026-04-20T07:40:57.440794077Z INFO: Gathering OVN_Southbound from ovnkube-node-qvqrn...
2026-04-20T07:40:57.441222750Z INFO: Gathering OVN_Southbound from ovnkube-node-vz4hl...
2026-04-20T07:40:57.442785310Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T07:40:57.728379736Z tar: Removing leading `/' from member names
2026-04-20T07:40:57.781963491Z Wrote inspect data to must-gather.
2026-04-20T07:40:57.869630326Z tar: Removing leading `/' from member names
2026-04-20T07:40:57.879354822Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-20T07:40:57.902381950Z Gathering data for ns/openshift-monitoring...
2026-04-20T07:40:57.915123120Z tar: Removing leading `/' from member names
2026-04-20T07:40:57.925235713Z tar: Removing leading `/' from member names
2026-04-20T07:40:57.961524597Z tar: Removing leading `/' from member names
2026-04-20T07:40:57.962858864Z tar: Removing leading `/' from member names
2026-04-20T07:40:58.176833098Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-20T07:40:58.240292126Z Wrote inspect data to must-gather.
2026-04-20T07:40:58.558470421Z INFO: Waiting for network log collection to complete ...
2026-04-20T07:40:58.558525629Z INFO: Waiting for ovnk database copies to complete ...
2026-04-20T07:40:58.559021558Z INFO: Copying ovnk databases complete.
2026-04-20T07:40:58.560886596Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-20T07:40:58.563551432Z ovnk_database_store/
2026-04-20T07:40:58.563624645Z ovnk_database_store/ovnkube-node-5bfnd_nbdb
2026-04-20T07:40:58.576506971Z ovnk_database_store/ovnkube-node-vz4hl_nbdb
2026-04-20T07:40:58.590107368Z ovnk_database_store/ovnkube-node-5bfnd_sbdb
2026-04-20T07:40:58.614386662Z ovnk_database_store/ovnkube-node-qvqrn_nbdb
2026-04-20T07:40:58.627208722Z ovnk_database_store/ovnkube-node-qvqrn_sbdb
2026-04-20T07:40:58.634690022Z Wrote inspect data to must-gather.
2026-04-20T07:40:58.646385236Z INFO: Getting status from alertmanager-main-0
2026-04-20T07:40:58.652112448Z ovnk_database_store/ovnkube-node-vz4hl_sbdb
2026-04-20T07:40:58.688074016Z INFO: Network log collection complete.
2026-04-20T07:40:58.973906024Z Wrote inspect data to must-gather.
2026-04-20T07:40:59.327768930Z Wrote inspect data to must-gather.
2026-04-20T07:40:59.681561325Z Wrote inspect data to must-gather.
2026-04-20T07:40:59.959869123Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-20T07:40:59.962094710Z INFO: Fallback to identify the container image from release info
2026-04-20T07:41:00.268227273Z Gathering data for ns/openshift-network-console...
2026-04-20T07:41:00.464180046Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T07:41:00.573292801Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-20T07:41:00.722171658Z daemonset.apps/perf-node-gather-daemonset created
2026-04-20T07:41:00.815027719Z Gathering data for ns/openshift-console-operator...
2026-04-20T07:41:00.871941156Z Waiting for performance profile collector pods to become ready: 1
2026-04-20T07:41:01.360104465Z Gathering data for ns/openshift-console...
2026-04-20T07:41:01.963000696Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-20T07:41:02.011218965Z Waiting for performance profile collector pods to become ready: 2
2026-04-20T07:41:02.406068830Z Gathering data for ns/openshift-dns-operator...
2026-04-20T07:41:02.670726930Z Wrote inspect data to must-gather.
2026-04-20T07:41:02.742138373Z Gathering data for ns/openshift-dns...
2026-04-20T07:41:03.129554901Z Waiting for performance profile collector pods to become ready: 3
2026-04-20T07:41:03.347689072Z Gathering data for ns/openshift-image-registry...
2026-04-20T07:41:03.836872357Z Gathering data for ns/openshift-ingress-operator...
2026-04-20T07:41:04.214920765Z Gathering data for ns/openshift-ingress...
2026-04-20T07:41:04.254938079Z Waiting for performance profile collector pods to become ready: 4
2026-04-20T07:41:05.022887923Z Gathering data for ns/openshift-ingress-canary...
2026-04-20T07:41:05.396496243Z Waiting for performance profile collector pods to become ready: 5
2026-04-20T07:41:05.469928026Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T07:41:05.510710176Z Gathering data for ns/openshift-insights...
2026-04-20T07:41:06.500593159Z Waiting for performance profile collector pods to become ready: 6
2026-04-20T07:41:07.485127910Z Gathering data for ns/opendatahub...
2026-04-20T07:41:07.619564308Z Waiting for performance profile collector pods to become ready: 7
2026-04-20T07:41:08.738624514Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-20T07:41:08.992033740Z Collecting performance related data for node ip-10-0-130-105.ec2.internal
2026-04-20T07:41:08.999187149Z Collecting performance related data for node ip-10-0-142-100.ec2.internal
2026-04-20T07:41:09.007437646Z Collecting performance related data for node ip-10-0-138-178.ec2.internal
2026-04-20T07:41:09.014896008Z Gathering data for ns/openshift-lws-operator...
2026-04-20T07:41:10.494566323Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-20T07:41:10.556016336Z Gathering data for ns/openshift-config...
2026-04-20T07:41:10.882003674Z Gathering data for ns/openshift-config-managed...
2026-04-20T07:41:11.354827770Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-20T07:41:11.752808744Z Gathering data for ns/openshift-kube-apiserver...
2026-04-20T07:41:12.098530723Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-20T07:41:12.448648149Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-20T07:41:12.784551336Z Gathering data for ns/openshift-kube-scheduler...
2026-04-20T07:41:13.108130970Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-20T07:41:13.464695051Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-20T07:41:13.848336980Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-20T07:41:14.220394794Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-20T07:41:14.832533119Z Gathering data for ns/openshift-multus...
2026-04-20T07:41:15.499882684Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-20T07:41:16.231379331Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-20T07:41:17.775054481Z Collecting kubelet logs for node ip-10-0-130-105.ec2.internal
2026-04-20T07:41:17.777155129Z Collecting kubelet logs for node ip-10-0-142-100.ec2.internal
2026-04-20T07:41:17.779447650Z Collecting kubelet logs for node ip-10-0-138-178.ec2.internal
2026-04-20T07:41:17.951378387Z Gathering data for ns/openshift-host-network...
2026-04-20T07:41:18.185005622Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-20T07:41:18.191877172Z INFO: Node performance data collection complete.
2026-04-20T07:41:18.373238418Z Gathering data for ns/openshift-network-diagnostics...
2026-04-20T07:41:18.938760442Z Gathering data for ns/openshift-network-node-identity...
2026-04-20T07:41:19.408403167Z Gathering data for ns/openshift-network-operator...
2026-04-20T07:41:19.813196080Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-20T07:41:20.108368835Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-20T07:41:20.505589923Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-20T07:41:20.627373601Z Gathering data for ns/openshift-apiserver-operator...
2026-04-20T07:41:20.973269857Z Gathering data for ns/openshift-apiserver...
2026-04-20T07:41:21.436095594Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-20T07:41:21.737710126Z Gathering data for ns/openshift-controller-manager...
2026-04-20T07:41:22.042298381Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-20T07:41:22.726503355Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-20T07:41:23.090019953Z Gathering data for ns/openshift-service-ca-operator...
2026-04-20T07:41:23.462440238Z Gathering data for ns/openshift-service-ca...
2026-04-20T07:41:23.888466149Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-20T07:41:24.962810717Z Wrote inspect data to must-gather.
2026-04-20T07:41:24.962858441Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T07:41:24.962858441Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-20T07:41:25.096409856Z error: the server doesn't have a resource type "clusters"
2026-04-20T07:41:25.288079955Z Caches written to disk