2026-04-20T15:03:06.416625487Z [disk usage checker] Started
2026-04-20T15:03:06.420349250Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:06.651706215Z Gathering data for ns/openshift-cluster-version...
2026-04-20T15:03:06.806229548Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T15:03:06.896299912Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T15:03:06.997673939Z Gathering data for ns/default...
2026-04-20T15:03:07.352031137Z Gathering data for ns/openshift...
2026-04-20T15:03:07.794972888Z Gathering data for ns/kube-system...
2026-04-20T15:03:08.611455762Z Gathering data for ns/openshift-etcd...
2026-04-20T15:03:08.842427112Z Waiting on subprocesses to finish execution.
2026-04-20T15:03:08.849603933Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-20T15:03:08.851534608Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-20T15:03:08.859726380Z INFO: Collecting host service logs for crio
2026-04-20T15:03:08.860215798Z INFO: Collecting host service logs for kubelet
2026-04-20T15:03:08.860584625Z INFO: Collecting host service logs for rpm-ostreed
2026-04-20T15:03:08.860899679Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-20T15:03:08.861233680Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-20T15:03:08.861581913Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-20T15:03:08.861922416Z INFO: Collecting host service logs for NetworkManager
2026-04-20T15:03:08.862279006Z INFO: Collecting host service logs for openvswitch
2026-04-20T15:03:08.862600238Z INFO: Collecting host service logs for ovs-configuration
2026-04-20T15:03:08.862938926Z INFO: Collecting host service logs for ovsdb-server
2026-04-20T15:03:08.863287981Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-20T15:03:08.863579077Z INFO: Gathering HAProxy config files
2026-04-20T15:03:08.863702509Z INFO: Waiting for worker host service log collection to complete ...
2026-04-20T15:03:08.875659251Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-20T15:03:08.978788175Z INFO: Waiting for node performance related collection to complete ...
2026-04-20T15:03:09.251983688Z Wrote inspect data to must-gather.
2026-04-20T15:03:09.252022878Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T15:03:09.252022878Z     namespaces "assisted-installer" not found
2026-04-20T15:03:09.477620422Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-20T15:03:09.484960161Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-20T15:03:09.647434740Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-20T15:03:09.683728178Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-20T15:03:09.969280232Z Wrote inspect data to must-gather.
2026-04-20T15:03:09.973920176Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-6hcxh
2026-04-20T15:03:09.973962905Z insights-runtime-extractor-2jkbs
2026-04-20T15:03:09.973982795Z insights-runtime-extractor-q2tq5
2026-04-20T15:03:09.974001879Z insights-runtime-extractor-v88gx
2026-04-20T15:03:10.091419418Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-20T15:03:10.091459034Z INFO: on-disk MachineConfig config collection complete.
2026-04-20T15:03:10.103921804Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-20T15:03:10.154049660Z No resources found in openshift-etcd namespace.
2026-04-20T15:03:10.344906421Z No resources found
2026-04-20T15:03:10.367605528Z INFO: "metallb-operator" not detected. Skipping.
2026-04-20T15:03:10.523006048Z INFO: Worker host service log collection to complete.
2026-04-20T15:03:10.751564176Z error: the server doesn't have a resource type "performanceprofile"
2026-04-20T15:03:10.990542163Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-20T15:03:10.990542163Z See 'oc rsync -h' for help and examples
2026-04-20T15:03:11.197811757Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T15:03:11.404924737Z ERROR: No running kube-apiserver pods found
2026-04-20T15:03:11.484093503Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:11.562675083Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-20T15:03:11.597090467Z Wrote inspect data to must-gather.
2026-04-20T15:03:11.620919688Z Wrote inspect data to must-gather.
2026-04-20T15:03:11.671337954Z Wrote inspect data to must-gather.
2026-04-20T15:03:11.676750347Z No resources found
2026-04-20T15:03:11.692018230Z Wrote inspect data to must-gather.
2026-04-20T15:03:11.711751599Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-20T15:03:11.742732221Z Wrote inspect data to must-gather.
2026-04-20T15:03:11.766037714Z INFO: 'previous-logs' folder not found on ip-10-0-139-136.ec2.internal, skipping...
2026-04-20T15:03:11.838564671Z INFO: Getting rules from prometheus-k8s-0
2026-04-20T15:03:11.993390229Z error: the server doesn't have a resource type "machineconfigs"
2026-04-20T15:03:12.010389578Z tar: Removing leading `/' from member names
2026-04-20T15:03:12.026830613Z INFO: HAProxy config collection complete.
2026-04-20T15:03:12.059656606Z Gathering data for ns/kuadrant-system...
2026-04-20T15:03:12.168183411Z INFO: 'previous-logs' folder not found on ip-10-0-140-30.ec2.internal, skipping...
2026-04-20T15:03:12.213424437Z Wrote inspect data to must-gather.
2026-04-20T15:03:12.313731150Z Wrote inspect data to must-gather.
2026-04-20T15:03:12.469362466Z INFO: Getting status/config from prometheus-k8s-0
2026-04-20T15:03:12.542472875Z INFO: 'previous-logs' folder not found on ip-10-0-142-166.ec2.internal, skipping...
2026-04-20T15:03:12.542590867Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-20T15:03:12.542649730Z INFO: Machine Config Daemon termination log collection complete.
2026-04-20T15:03:12.647988728Z Wrote inspect data to must-gather.
2026-04-20T15:03:12.672841512Z Wrote inspect data to must-gather.
2026-04-20T15:03:12.878251298Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-20T15:03:13.020623809Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-20T15:03:13.024403853Z Wrote inspect data to must-gather.
2026-04-20T15:03:13.222053377Z Wrote inspect data to must-gather.
2026-04-20T15:03:13.259092926Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-20T15:03:13.314633203Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-20T15:03:13.429641354Z INFO: INTERCONNECT MODE
2026-04-20T15:03:13.429677388Z INFO: Gathering ovn-kubernetes DBs
2026-04-20T15:03:13.534867667Z Wrote inspect data to must-gather.
2026-04-20T15:03:13.604979017Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-20T15:03:13.659072819Z INFO: Gathering OVN_Northbound from ovnkube-node-r8xsm...
2026-04-20T15:03:13.659360442Z INFO: Gathering OVN_Northbound from ovnkube-node-s4vzx...
2026-04-20T15:03:13.659582689Z INFO: Gathering OVN_Northbound from ovnkube-node-x29sq...
2026-04-20T15:03:13.659869868Z INFO: Gathering OVN_Southbound from ovnkube-node-r8xsm...
2026-04-20T15:03:13.660102203Z INFO: Gathering OVN_Southbound from ovnkube-node-s4vzx...
2026-04-20T15:03:13.660355033Z INFO: Gathering OVN_Southbound from ovnkube-node-x29sq...
2026-04-20T15:03:14.027571413Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-20T15:03:14.097870826Z tar: Removing leading `/' from member names
2026-04-20T15:03:14.118134419Z tar: Removing leading `/' from member names
2026-04-20T15:03:14.176205059Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-20T15:03:14.189285932Z tar: Removing leading `/' from member names
2026-04-20T15:03:14.196764557Z tar: Removing leading `/' from member names
2026-04-20T15:03:14.203288201Z tar: Removing leading `/' from member names
2026-04-20T15:03:14.215311319Z tar: Removing leading `/' from member names
2026-04-20T15:03:14.242528118Z Wrote inspect data to must-gather.
2026-04-20T15:03:14.315258977Z Gathering data for ns/openshift-monitoring...
2026-04-20T15:03:14.656184426Z Wrote inspect data to must-gather.
2026-04-20T15:03:14.664558444Z INFO: Getting status from alertmanager-main-0
2026-04-20T15:03:14.782391529Z INFO: Waiting for network log collection to complete ...
2026-04-20T15:03:14.782508754Z INFO: Waiting for ovnk database copies to complete ...
2026-04-20T15:03:14.783170483Z INFO: Copying ovnk databases complete.
2026-04-20T15:03:14.785300932Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-20T15:03:14.788385193Z ovnk_database_store/
2026-04-20T15:03:14.788450388Z ovnk_database_store/ovnkube-node-x29sq_nbdb
2026-04-20T15:03:14.802890396Z ovnk_database_store/ovnkube-node-s4vzx_sbdb
2026-04-20T15:03:14.830377553Z ovnk_database_store/ovnkube-node-s4vzx_nbdb
2026-04-20T15:03:14.841009062Z ovnk_database_store/ovnkube-node-r8xsm_sbdb
2026-04-20T15:03:14.869458909Z ovnk_database_store/ovnkube-node-x29sq_sbdb
2026-04-20T15:03:14.890183139Z ovnk_database_store/ovnkube-node-r8xsm_nbdb
2026-04-20T15:03:14.903954963Z INFO: Network log collection complete.
2026-04-20T15:03:15.025314245Z Wrote inspect data to must-gather.
2026-04-20T15:03:15.313281426Z Wrote inspect data to must-gather.
2026-04-20T15:03:15.630294498Z Wrote inspect data to must-gather.
2026-04-20T15:03:15.928057762Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-20T15:03:15.930518595Z INFO: Fallback to identify the container image from release info
2026-04-20T15:03:16.489930430Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:16.605750096Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-20T15:03:16.636653807Z Gathering data for ns/openshift-network-console...
2026-04-20T15:03:16.753310419Z daemonset.apps/perf-node-gather-daemonset created
2026-04-20T15:03:16.907350705Z Waiting for performance profile collector pods to become ready: 1
2026-04-20T15:03:17.216794351Z Gathering data for ns/openshift-console-operator...
2026-04-20T15:03:17.694751893Z Gathering data for ns/openshift-console...
2026-04-20T15:03:17.951389723Z Wrote inspect data to must-gather.
2026-04-20T15:03:18.017835322Z Waiting for performance profile collector pods to become ready: 2
2026-04-20T15:03:18.236322328Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-20T15:03:18.638951230Z Gathering data for ns/openshift-dns-operator...
2026-04-20T15:03:18.968548043Z Gathering data for ns/openshift-dns...
2026-04-20T15:03:19.121731115Z Waiting for performance profile collector pods to become ready: 3
2026-04-20T15:03:19.713834431Z Gathering data for ns/openshift-image-registry...
2026-04-20T15:03:20.215654915Z Gathering data for ns/openshift-ingress-operator...
2026-04-20T15:03:20.260447806Z Waiting for performance profile collector pods to become ready: 4
2026-04-20T15:03:20.735136647Z Gathering data for ns/openshift-ingress...
2026-04-20T15:03:21.390343030Z Waiting for performance profile collector pods to become ready: 5
2026-04-20T15:03:21.496589114Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:21.538924791Z Gathering data for ns/openshift-ingress-canary...
2026-04-20T15:03:21.999552330Z Gathering data for ns/openshift-insights...
2026-04-20T15:03:22.535998289Z Waiting for performance profile collector pods to become ready: 6
2026-04-20T15:03:23.646253917Z Waiting for performance profile collector pods to become ready: 7
2026-04-20T15:03:24.321411068Z Gathering data for ns/opendatahub...
2026-04-20T15:03:24.753801196Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-20T15:03:25.001882871Z Collecting performance related data for node ip-10-0-142-166.ec2.internal
2026-04-20T15:03:25.015239320Z Collecting performance related data for node ip-10-0-140-30.ec2.internal
2026-04-20T15:03:25.021197923Z Collecting performance related data for node ip-10-0-139-136.ec2.internal
2026-04-20T15:03:26.051483840Z Gathering data for ns/openshift-lws-operator...
2026-04-20T15:03:26.502446416Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:27.462716835Z Gathering data for ns/openshift-config...
2026-04-20T15:03:27.853611541Z Gathering data for ns/openshift-config-managed...
2026-04-20T15:03:28.379179478Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-20T15:03:28.758632414Z Gathering data for ns/openshift-kube-apiserver...
2026-04-20T15:03:29.090315042Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-20T15:03:29.452143845Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-20T15:03:29.806526074Z Gathering data for ns/openshift-kube-scheduler...
2026-04-20T15:03:30.126418508Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-20T15:03:30.501531038Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-20T15:03:30.888648834Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-20T15:03:31.252034959Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-20T15:03:31.509855951Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:31.943721508Z Gathering data for ns/openshift-multus...
2026-04-20T15:03:33.484901145Z Collecting kubelet logs for node ip-10-0-142-166.ec2.internal
2026-04-20T15:03:33.486932146Z Collecting kubelet logs for node ip-10-0-140-30.ec2.internal
2026-04-20T15:03:33.489745611Z Collecting kubelet logs for node ip-10-0-139-136.ec2.internal
2026-04-20T15:03:33.532431022Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-20T15:03:33.858529261Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-20T15:03:33.863581223Z INFO: Node performance data collection complete.
2026-04-20T15:03:35.104739077Z Gathering data for ns/openshift-host-network...
2026-04-20T15:03:35.469657177Z Gathering data for ns/openshift-network-diagnostics...
2026-04-20T15:03:36.104596715Z Gathering data for ns/openshift-network-node-identity...
2026-04-20T15:03:36.517158354Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:36.602220498Z Gathering data for ns/openshift-network-operator...
2026-04-20T15:03:37.017327311Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-20T15:03:37.321764271Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-20T15:03:37.825744199Z Gathering data for ns/openshift-apiserver-operator...
2026-04-20T15:03:38.163353860Z Gathering data for ns/openshift-apiserver...
2026-04-20T15:03:38.601350196Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-20T15:03:38.981303068Z Gathering data for ns/openshift-controller-manager...
2026-04-20T15:03:39.293503443Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-20T15:03:39.968023056Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-20T15:03:40.361145273Z Gathering data for ns/openshift-service-ca-operator...
2026-04-20T15:03:40.745354237Z Gathering data for ns/openshift-service-ca...
2026-04-20T15:03:41.308084951Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-20T15:03:41.521998901Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-20T15:03:42.406891436Z Wrote inspect data to must-gather.
2026-04-20T15:03:42.406944553Z error: inspection completed with the errors occurred while gathering data:
2026-04-20T15:03:42.406944553Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-20T15:03:42.549331725Z error: the server doesn't have a resource type "clusters"
2026-04-20T15:03:42.772240435Z Caches written to disk