2026-04-28T20:38:11.901488138Z [disk usage checker] Started
2026-04-28T20:38:11.932206851Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-28T20:38:12.118080696Z Gathering data for ns/openshift-cluster-version...
2026-04-28T20:38:12.268608099Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-28T20:38:12.358327951Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-28T20:38:12.450702622Z Gathering data for ns/default...
2026-04-28T20:38:12.793391045Z Gathering data for ns/openshift...
2026-04-28T20:38:13.240180373Z Gathering data for ns/kube-system...
2026-04-28T20:38:13.988600779Z Gathering data for ns/openshift-etcd...
2026-04-28T20:38:14.332635282Z Wrote inspect data to must-gather.
2026-04-28T20:38:14.332667224Z error: inspection completed with the errors occurred while gathering data:
2026-04-28T20:38:14.332667224Z     namespaces "assisted-installer" not found
2026-04-28T20:38:14.426901927Z Waiting on subprocesses to finish execution.
2026-04-28T20:38:14.433939150Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-28T20:38:14.445874881Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-28T20:38:14.451581725Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-28T20:38:14.451912952Z INFO: Waiting for node performance related collection to complete ...
2026-04-28T20:38:14.462105532Z INFO: Collecting host service logs for crio
2026-04-28T20:38:14.462499664Z INFO: Collecting host service logs for kubelet
2026-04-28T20:38:14.462882303Z INFO: Collecting host service logs for rpm-ostreed
2026-04-28T20:38:14.463237766Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-28T20:38:14.463622996Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-28T20:38:14.463976473Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-28T20:38:14.464340116Z INFO: Collecting host service logs for NetworkManager
2026-04-28T20:38:14.464762281Z INFO: Collecting host service logs for openvswitch
2026-04-28T20:38:14.465123868Z INFO: Collecting host service logs for ovs-configuration
2026-04-28T20:38:14.465492789Z INFO: Collecting host service logs for ovsdb-server
2026-04-28T20:38:14.465854091Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-28T20:38:14.466268687Z INFO: Waiting for worker host service log collection to complete ...
2026-04-28T20:38:14.480146849Z INFO: Gathering HAProxy config files
2026-04-28T20:38:15.401065919Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-x7kx7
2026-04-28T20:38:15.401104841Z insights-runtime-extractor-9zfhp
2026-04-28T20:38:15.401122934Z insights-runtime-extractor-fjrcg
2026-04-28T20:38:15.401140168Z insights-runtime-extractor-nghpr
2026-04-28T20:38:15.508963304Z error: the server doesn't have a resource type "performanceprofile"
2026-04-28T20:38:15.563144859Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-28T20:38:15.563211789Z INFO: on-disk MachineConfig config collection complete.
2026-04-28T20:38:15.603177598Z No resources found in openshift-etcd namespace.
2026-04-28T20:38:15.759229746Z No resources found
2026-04-28T20:38:15.777832001Z INFO: "metallb-operator" not detected. Skipping.
2026-04-28T20:38:15.840736150Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-28T20:38:15.840736150Z See 'oc rsync -h' for help and examples
2026-04-28T20:38:15.850679764Z Wrote inspect data to must-gather.
2026-04-28T20:38:15.930810832Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-28T20:38:15.934094711Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-28T20:38:16.009145086Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-28T20:38:16.016078109Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-28T20:38:16.113789361Z INFO: Worker host service log collection to complete.
2026-04-28T20:38:16.313986240Z ERROR: No running kube-apiserver pods found
2026-04-28T20:38:16.357354813Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-28T20:38:16.522825961Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-28T20:38:16.632352266Z No resources found
2026-04-28T20:38:16.798755563Z Wrote inspect data to must-gather.
2026-04-28T20:38:16.853195794Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-28T20:38:17.033283252Z Wrote inspect data to must-gather.
2026-04-28T20:38:17.038102452Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-28T20:38:17.116992835Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-28T20:38:17.167746257Z INFO: 'previous-logs' folder not found on ip-10-0-131-5.ec2.internal, skipping...
2026-04-28T20:38:17.198754804Z Gathering data for ns/kuadrant-system...
2026-04-28T20:38:17.231630277Z tar: Removing leading `/' from member names
2026-04-28T20:38:17.243915057Z INFO: HAProxy config collection complete.
2026-04-28T20:38:17.258192778Z Wrote inspect data to must-gather.
2026-04-28T20:38:17.403352032Z Wrote inspect data to must-gather.
2026-04-28T20:38:17.411573484Z error: the server doesn't have a resource type "machineconfigs"
2026-04-28T20:38:17.532800664Z Wrote inspect data to must-gather.
2026-04-28T20:38:17.636608987Z INFO: 'previous-logs' folder not found on ip-10-0-141-41.ec2.internal, skipping...
2026-04-28T20:38:17.803681523Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-28T20:38:17.813837300Z Wrote inspect data to must-gather.
2026-04-28T20:38:17.814971973Z INFO: Getting rules from prometheus-k8s-0
2026-04-28T20:38:17.841357802Z Wrote inspect data to must-gather.
2026-04-28T20:38:18.058033745Z INFO: 'previous-logs' folder not found on ip-10-0-143-22.ec2.internal, skipping...
2026-04-28T20:38:18.058132454Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-28T20:38:18.058218878Z INFO: Machine Config Daemon termination log collection complete.
2026-04-28T20:38:18.145716541Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-28T20:38:18.218656637Z Wrote inspect data to must-gather.
2026-04-28T20:38:18.296850205Z Wrote inspect data to must-gather.
2026-04-28T20:38:18.350633411Z INFO: Getting status/config from prometheus-k8s-0
2026-04-28T20:38:18.383078468Z Gathering data for ns/openshift-monitoring...
2026-04-28T20:38:18.446757336Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-28T20:38:18.673135328Z Wrote inspect data to must-gather.
2026-04-28T20:38:18.766448021Z Wrote inspect data to must-gather.
2026-04-28T20:38:18.845179087Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-28T20:38:19.043818298Z INFO: INTERCONNECT MODE
2026-04-28T20:38:19.043848760Z INFO: Gathering ovn-kubernetes DBs
2026-04-28T20:38:19.079033825Z Wrote inspect data to must-gather.
2026-04-28T20:38:19.101005126Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-28T20:38:19.233457289Z INFO: Gathering OVN_Northbound from ovnkube-node-pqj2n...
2026-04-28T20:38:19.233663259Z INFO: Gathering OVN_Northbound from ovnkube-node-ptjf5...
2026-04-28T20:38:19.233901857Z INFO: Gathering OVN_Northbound from ovnkube-node-ssm92...
2026-04-28T20:38:19.234436006Z INFO: Gathering OVN_Southbound from ovnkube-node-pqj2n...
2026-04-28T20:38:19.234709780Z INFO: Gathering OVN_Southbound from ovnkube-node-ptjf5...
2026-04-28T20:38:19.235004848Z INFO: Gathering OVN_Southbound from ovnkube-node-ssm92...
2026-04-28T20:38:19.412664453Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-28T20:38:19.496504006Z tar: Removing leading `/' from member names
2026-04-28T20:38:19.513012696Z tar: Removing leading `/' from member names
2026-04-28T20:38:19.565647958Z Wrote inspect data to must-gather.
2026-04-28T20:38:19.683577255Z tar: Removing leading `/' from member names
2026-04-28T20:38:19.698165187Z tar: Removing leading `/' from member names
2026-04-28T20:38:19.706340083Z tar: Removing leading `/' from member names
2026-04-28T20:38:19.709744083Z tar: Removing leading `/' from member names
2026-04-28T20:38:19.816845542Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-28T20:38:19.942015990Z Wrote inspect data to must-gather.
2026-04-28T20:38:20.253190752Z INFO: Waiting for network log collection to complete ...
2026-04-28T20:38:20.253241260Z INFO: Waiting for ovnk database copies to complete ...
2026-04-28T20:38:20.253751974Z INFO: Copying ovnk databases complete.
2026-04-28T20:38:20.255316113Z 5.1M	must-gather/network_logs/ovnk_database_store
2026-04-28T20:38:20.257693546Z ovnk_database_store/
2026-04-28T20:38:20.257693546Z ovnk_database_store/ovnkube-node-ssm92_nbdb
2026-04-28T20:38:20.270300831Z ovnk_database_store/ovnkube-node-pqj2n_nbdb
2026-04-28T20:38:20.274233212Z INFO: Getting status from alertmanager-main-0
2026-04-28T20:38:20.283865395Z ovnk_database_store/ovnkube-node-ssm92_sbdb
2026-04-28T20:38:20.295503575Z Wrote inspect data to must-gather.
2026-04-28T20:38:20.318947549Z ovnk_database_store/ovnkube-node-ptjf5_nbdb
2026-04-28T20:38:20.332800805Z ovnk_database_store/ovnkube-node-ptjf5_sbdb
2026-04-28T20:38:20.369237924Z ovnk_database_store/ovnkube-node-pqj2n_sbdb
2026-04-28T20:38:20.410099721Z INFO: Network log collection complete.
2026-04-28T20:38:20.668113195Z Wrote inspect data to must-gather.
2026-04-28T20:38:20.755802491Z Gathering data for ns/openshift-network-console...
2026-04-28T20:38:21.039888387Z Wrote inspect data to must-gather.
2026-04-28T20:38:21.277681863Z Gathering data for ns/openshift-console-operator...
2026-04-28T20:38:21.363148418Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-28T20:38:21.366190055Z INFO: Fallback to identify the container image from release info
2026-04-28T20:38:21.701222951Z Gathering data for ns/openshift-console...
2026-04-28T20:38:21.923073076Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-28T20:38:22.074441017Z daemonset.apps/perf-node-gather-daemonset created
2026-04-28T20:38:22.112249103Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-28T20:38:22.237131961Z Waiting for performance profile collector pods to become ready: 1
2026-04-28T20:38:22.244891370Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-28T20:38:22.338556279Z Wrote inspect data to must-gather.
2026-04-28T20:38:22.626929084Z Gathering data for ns/openshift-dns-operator...
2026-04-28T20:38:22.933340271Z Gathering data for ns/openshift-dns...
2026-04-28T20:38:23.374238482Z Waiting for performance profile collector pods to become ready: 2
2026-04-28T20:38:23.483846592Z Gathering data for ns/openshift-image-registry...
2026-04-28T20:38:23.958036761Z Gathering data for ns/openshift-ingress-operator...
2026-04-28T20:38:24.288945406Z Gathering data for ns/openshift-ingress...
2026-04-28T20:38:24.518840743Z Waiting for performance profile collector pods to become ready: 3
2026-04-28T20:38:24.821497483Z Gathering data for ns/openshift-ingress-canary...
2026-04-28T20:38:25.238565409Z Gathering data for ns/openshift-insights...
2026-04-28T20:38:25.648019098Z Waiting for performance profile collector pods to become ready: 4
2026-04-28T20:38:26.765895173Z Waiting for performance profile collector pods to become ready: 5
2026-04-28T20:38:27.116850915Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 70
2026-04-28T20:38:27.882799889Z Waiting for performance profile collector pods to become ready: 6
2026-04-28T20:38:27.888692074Z Gathering data for ns/openshift-lws-operator...
2026-04-28T20:38:28.408584854Z Gathering data for ns/kserve...
2026-04-28T20:38:29.012919237Z Waiting for performance profile collector pods to become ready: 7
2026-04-28T20:38:30.110465812Z Gathering data for ns/openshift-config...
2026-04-28T20:38:30.148932486Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-28T20:38:30.417881886Z Gathering data for ns/openshift-config-managed...
2026-04-28T20:38:30.432117411Z Collecting performance related data for node ip-10-0-131-5.ec2.internal
2026-04-28T20:38:30.439691816Z Collecting performance related data for node ip-10-0-143-22.ec2.internal
2026-04-28T20:38:30.451612152Z Collecting performance related data for node ip-10-0-141-41.ec2.internal
2026-04-28T20:38:30.969692167Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-28T20:38:31.375781847Z Gathering data for ns/openshift-kube-apiserver...
2026-04-28T20:38:31.742619912Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-28T20:38:32.122153927Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-04-28T20:38:32.126682265Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-28T20:38:32.420515083Z Gathering data for ns/openshift-kube-scheduler...
2026-04-28T20:38:32.745904841Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-28T20:38:33.082722556Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-28T20:38:33.458812390Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-28T20:38:33.807034231Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-28T20:38:34.345967218Z Gathering data for ns/openshift-multus...
2026-04-28T20:38:35.667983649Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-28T20:38:37.127528984Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-04-28T20:38:37.245926908Z Gathering data for ns/openshift-host-network...
2026-04-28T20:38:37.582834166Z Gathering data for ns/openshift-network-diagnostics...
2026-04-28T20:38:38.139854633Z Gathering data for ns/openshift-network-node-identity...
2026-04-28T20:38:38.560864369Z Gathering data for ns/openshift-network-operator...
2026-04-28T20:38:38.927158471Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-28T20:38:39.053869876Z Collecting kubelet logs for node ip-10-0-131-5.ec2.internal
2026-04-28T20:38:39.055594973Z Collecting kubelet logs for node ip-10-0-143-22.ec2.internal
2026-04-28T20:38:39.057224405Z Collecting kubelet logs for node ip-10-0-141-41.ec2.internal
2026-04-28T20:38:39.269480021Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-28T20:38:39.481618547Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-28T20:38:39.485844685Z INFO: Node performance data collection complete.
2026-04-28T20:38:39.779943764Z Gathering data for ns/openshift-apiserver-operator...
2026-04-28T20:38:40.081120373Z Gathering data for ns/openshift-apiserver...
2026-04-28T20:38:40.493040769Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-28T20:38:40.755192908Z Gathering data for ns/openshift-controller-manager...
2026-04-28T20:38:41.060489627Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-28T20:38:41.720970703Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-28T20:38:42.004987942Z Gathering data for ns/openshift-service-ca-operator...
2026-04-28T20:38:42.132350971Z [disk usage checker] Volume usage percentage: current = 20 ; allowed = 70
2026-04-28T20:38:42.321721508Z Gathering data for ns/openshift-service-ca...
2026-04-28T20:38:42.687152200Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-28T20:38:43.559265385Z Wrote inspect data to must-gather.
2026-04-28T20:38:43.559307820Z error: inspection completed with the errors occurred while gathering data:
2026-04-28T20:38:43.559307820Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-28T20:38:43.694696458Z error: the server doesn't have a resource type "clusters"
2026-04-28T20:38:44.128500771Z Caches written to disk