2026-04-16T19:56:01.368915117Z [disk usage checker] Started
2026-04-16T19:56:01.403974521Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:56:01.607379680Z Gathering data for ns/openshift-cluster-version...
2026-04-16T19:56:01.770648837Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:56:01.869099903Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:56:01.968544285Z Gathering data for ns/default...
2026-04-16T19:56:02.340876951Z Gathering data for ns/openshift...
2026-04-16T19:56:02.818822775Z Gathering data for ns/kube-system...
2026-04-16T19:56:03.568275579Z Gathering data for ns/openshift-etcd...
2026-04-16T19:56:03.914667802Z Waiting on subprocesses to finish execution.
2026-04-16T19:56:03.921597300Z INFO: Collecting host service logs for crio
2026-04-16T19:56:03.921913988Z INFO: Collecting host service logs for kubelet
2026-04-16T19:56:03.922329207Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T19:56:03.922665984Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T19:56:03.923018936Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T19:56:03.923391301Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T19:56:03.923746581Z INFO: Collecting host service logs for NetworkManager
2026-04-16T19:56:03.924127177Z INFO: Collecting host service logs for openvswitch
2026-04-16T19:56:03.924624812Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T19:56:03.924977429Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T19:56:03.925325517Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T19:56:03.925695251Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T19:56:03.926237756Z Wrote inspect data to must-gather.
2026-04-16T19:56:03.926316551Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:56:03.926316551Z     namespaces "assisted-installer" not found
2026-04-16T19:56:03.928555673Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T19:56:03.938073044Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T19:56:03.940684842Z INFO: Gathering HAProxy config files
2026-04-16T19:56:03.955107058Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T19:56:03.962172982Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T19:56:04.648626956Z No resources found in openshift-etcd namespace.
2026-04-16T19:56:04.812641001Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T19:56:04.818912627Z No resources found
2026-04-16T19:56:04.835376303Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-ksczz
2026-04-16T19:56:04.835376303Z insights-runtime-extractor-6sxn2
2026-04-16T19:56:04.835376303Z insights-runtime-extractor-fhd58
2026-04-16T19:56:04.835376303Z insights-runtime-extractor-qr8c4
2026-04-16T19:56:04.979320928Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T19:56:05.036753585Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T19:56:05.056103054Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T19:56:05.140442273Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T19:56:05.140551648Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T19:56:05.159269825Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T19:56:05.199267022Z INFO: Worker host service log collection to complete.
2026-04-16T19:56:05.402596094Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T19:56:05.510959737Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T19:56:05.518576448Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T19:56:05.968467157Z ERROR: No running kube-apiserver pods found
2026-04-16T19:56:05.968795293Z Wrote inspect data to must-gather.
2026-04-16T19:56:06.109906750Z Wrote inspect data to must-gather.
2026-04-16T19:56:06.178300948Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T19:56:06.178300948Z See 'oc rsync -h' for help and examples
2026-04-16T19:56:06.260304633Z No resources found
2026-04-16T19:56:06.279840137Z tar: Removing leading `/' from member names
2026-04-16T19:56:06.285986584Z INFO: HAProxy config collection complete.
2026-04-16T19:56:06.381083267Z INFO: 'previous-logs' folder not found on ip-10-0-128-123.ec2.internal, skipping...
2026-04-16T19:56:06.418978592Z Wrote inspect data to must-gather.
2026-04-16T19:56:06.544494374Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T19:56:06.597840504Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:56:06.614882254Z Wrote inspect data to must-gather.
2026-04-16T19:56:06.640604152Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:56:06.675960147Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T19:56:06.724530780Z Wrote inspect data to must-gather.
2026-04-16T19:56:06.745471875Z Wrote inspect data to must-gather.
2026-04-16T19:56:06.859691624Z INFO: 'previous-logs' folder not found on ip-10-0-130-163.ec2.internal, skipping...
2026-04-16T19:56:06.933111237Z Gathering data for ns/kuadrant-system...
2026-04-16T19:56:07.032940219Z Wrote inspect data to must-gather.
2026-04-16T19:56:07.099179486Z Wrote inspect data to must-gather.
2026-04-16T19:56:07.156695918Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T19:56:07.253411252Z INFO: 'previous-logs' folder not found on ip-10-0-130-83.ec2.internal, skipping...
2026-04-16T19:56:07.253520671Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T19:56:07.253569230Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T19:56:07.374615857Z Wrote inspect data to must-gather.
2026-04-16T19:56:07.494375859Z Wrote inspect data to must-gather.
2026-04-16T19:56:07.614327966Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T19:56:07.639002791Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T19:56:07.827361335Z Wrote inspect data to must-gather.
2026-04-16T19:56:07.954225620Z Wrote inspect data to must-gather.
2026-04-16T19:56:08.037063333Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:56:08.129977883Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T19:56:08.245160196Z INFO: INTERCONNECT MODE
2026-04-16T19:56:08.245236891Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T19:56:08.278977969Z Wrote inspect data to must-gather.
2026-04-16T19:56:08.411412895Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T19:56:08.460696202Z INFO: Gathering OVN_Northbound from ovnkube-node-4xbfq...
2026-04-16T19:56:08.461072884Z INFO: Gathering OVN_Northbound from ovnkube-node-bcplr...
2026-04-16T19:56:08.461423123Z INFO: Gathering OVN_Northbound from ovnkube-node-q4qdv...
2026-04-16T19:56:08.461765992Z INFO: Gathering OVN_Southbound from ovnkube-node-4xbfq...
2026-04-16T19:56:08.462045801Z INFO: Gathering OVN_Southbound from ovnkube-node-bcplr...
2026-04-16T19:56:08.462467988Z INFO: Gathering OVN_Southbound from ovnkube-node-q4qdv...
2026-04-16T19:56:08.655490255Z W0416 19:56:08.655426    1215 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T19:56:08.688678039Z Wrote inspect data to must-gather.
2026-04-16T19:56:08.723400428Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:56:08.850183669Z tar: Removing leading `/' from member names
2026-04-16T19:56:08.850373635Z tar: Removing leading `/' from member names
2026-04-16T19:56:08.869524296Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T19:56:08.905653910Z tar: Removing leading `/' from member names
2026-04-16T19:56:08.940810882Z tar: Removing leading `/' from member names
2026-04-16T19:56:08.945868455Z tar: Removing leading `/' from member names
2026-04-16T19:56:08.967650469Z tar: Removing leading `/' from member names
2026-04-16T19:56:09.085940814Z Gathering data for ns/openshift-monitoring...
2026-04-16T19:56:09.204002607Z Wrote inspect data to must-gather.
2026-04-16T19:56:09.220082792Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T19:56:09.593531643Z Wrote inspect data to must-gather.
2026-04-16T19:56:09.648360773Z INFO: Waiting for network log collection to complete ...
2026-04-16T19:56:09.648390839Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T19:56:09.648778952Z INFO: Copying ovnk databases complete.
2026-04-16T19:56:09.650501554Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-16T19:56:09.653126142Z ovnk_database_store/
2026-04-16T19:56:09.653126142Z ovnk_database_store/ovnkube-node-bcplr_nbdb
2026-04-16T19:56:09.669588174Z ovnk_database_store/ovnkube-node-bcplr_sbdb
2026-04-16T19:56:09.701492664Z ovnk_database_store/ovnkube-node-4xbfq_sbdb
2026-04-16T19:56:09.709701236Z INFO: Getting status from alertmanager-main-0
2026-04-16T19:56:09.731539763Z ovnk_database_store/ovnkube-node-q4qdv_sbdb
2026-04-16T19:56:09.759024063Z ovnk_database_store/ovnkube-node-q4qdv_nbdb
2026-04-16T19:56:09.769248538Z ovnk_database_store/ovnkube-node-4xbfq_nbdb
2026-04-16T19:56:09.783769340Z INFO: Network log collection complete.
2026-04-16T19:56:09.956116269Z Wrote inspect data to must-gather.
2026-04-16T19:56:10.234358853Z Wrote inspect data to must-gather.
2026-04-16T19:56:10.535785698Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T19:56:10.538722925Z INFO: Fallback to identify the container image from release info
2026-04-16T19:56:11.265596275Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-16T19:56:11.268204704Z Gathering data for ns/openshift-network-console...
2026-04-16T19:56:11.412662252Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T19:56:11.545142759Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T19:56:11.617086444Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:56:11.864348878Z Gathering data for ns/openshift-console-operator...
2026-04-16T19:56:12.372627264Z Gathering data for ns/openshift-console...
2026-04-16T19:56:12.681087774Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T19:56:12.889871194Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T19:56:13.280082609Z Wrote inspect data to must-gather.
2026-04-16T19:56:13.328268181Z Gathering data for ns/openshift-dns-operator...
2026-04-16T19:56:13.671340134Z Gathering data for ns/openshift-dns...
2026-04-16T19:56:13.791030266Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T19:56:14.302492895Z Gathering data for ns/openshift-image-registry...
2026-04-16T19:56:14.799799024Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T19:56:14.925489321Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T19:56:15.186876804Z Gathering data for ns/openshift-ingress...
2026-04-16T19:56:15.899319163Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T19:56:16.056811176Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T19:56:16.376520863Z Gathering data for ns/openshift-insights...
2026-04-16T19:56:16.621876538Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:56:17.173251390Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T19:56:18.312452005Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T19:56:18.406015242Z Gathering data for ns/opendatahub...
2026-04-16T19:56:19.437213296Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T19:56:19.688217131Z Collecting performance related data for node ip-10-0-130-163.ec2.internal
2026-04-16T19:56:19.699951752Z Collecting performance related data for node ip-10-0-130-83.ec2.internal
2026-04-16T19:56:19.709722404Z Collecting performance related data for node ip-10-0-128-123.ec2.internal
2026-04-16T19:56:19.932901612Z Gathering data for ns/openshift-lws-operator...
2026-04-16T19:56:21.389855882Z Gathering data for ns/openshift-config...
2026-04-16T19:56:21.627515226Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:56:21.689661382Z Gathering data for ns/openshift-config-managed...
2026-04-16T19:56:22.263885128Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T19:56:22.645469716Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T19:56:23.045309333Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T19:56:23.411479411Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T19:56:23.796948030Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T19:56:24.120872937Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T19:56:24.504508817Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T19:56:24.885204487Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T19:56:25.256811040Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T19:56:25.856521581Z Gathering data for ns/openshift-multus...
2026-04-16T19:56:26.633583653Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:56:27.324086619Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T19:56:28.239295378Z Collecting kubelet logs for node ip-10-0-130-163.ec2.internal
2026-04-16T19:56:28.241090249Z Collecting kubelet logs for node ip-10-0-130-83.ec2.internal
2026-04-16T19:56:28.243388143Z Collecting kubelet logs for node ip-10-0-128-123.ec2.internal
2026-04-16T19:56:28.613961323Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T19:56:28.619877583Z INFO: Node performance data collection complete.
2026-04-16T19:56:28.919947674Z Gathering data for ns/openshift-host-network...
2026-04-16T19:56:29.325448187Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T19:56:29.910769538Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T19:56:30.424966026Z Gathering data for ns/openshift-network-operator...
2026-04-16T19:56:30.836564116Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T19:56:31.158131839Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T19:56:31.632127015Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T19:56:31.638110225Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:56:31.985337984Z Gathering data for ns/openshift-apiserver...
2026-04-16T19:56:32.430399920Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T19:56:32.750743682Z Gathering data for ns/openshift-controller-manager...
2026-04-16T19:56:33.100971224Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T19:56:33.804005479Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T19:56:34.143243158Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T19:56:34.482949399Z Gathering data for ns/openshift-service-ca...
2026-04-16T19:56:34.889254055Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T19:56:35.897473186Z Wrote inspect data to must-gather.
2026-04-16T19:56:35.897661947Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:56:35.897661947Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T19:56:36.034026662Z error: the server doesn't have a resource type "clusters"
2026-04-16T19:56:36.231645540Z Caches written to disk