2026-05-20T14:45:24.551135757Z [disk usage checker] Started
2026-05-20T14:45:24.554828940Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:24.794556154Z Gathering data for ns/openshift-cluster-version...
2026-05-20T14:45:24.964678475Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-20T14:45:25.085925782Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T14:45:25.195448520Z Gathering data for ns/default...
2026-05-20T14:45:25.625240340Z Gathering data for ns/openshift...
2026-05-20T14:45:26.145267987Z Gathering data for ns/kube-system...
2026-05-20T14:45:27.016601296Z Gathering data for ns/openshift-etcd...
2026-05-20T14:45:27.148154895Z Waiting on subprocesses to finish execution.
2026-05-20T14:45:27.158812153Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-20T14:45:27.163608041Z INFO: Gathering HAProxy config files
2026-05-20T14:45:27.167194012Z INFO: Collecting host service logs for crio
2026-05-20T14:45:27.167669314Z INFO: Collecting host service logs for kubelet
2026-05-20T14:45:27.168022932Z INFO: Collecting host service logs for rpm-ostreed
2026-05-20T14:45:27.168359852Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-20T14:45:27.168717794Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-20T14:45:27.169063001Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-20T14:45:27.169369828Z INFO: Collecting host service logs for NetworkManager
2026-05-20T14:45:27.169774787Z INFO: Collecting host service logs for openvswitch
2026-05-20T14:45:27.170134914Z INFO: Collecting host service logs for ovs-configuration
2026-05-20T14:45:27.170471726Z INFO: Collecting host service logs for ovsdb-server
2026-05-20T14:45:27.170829148Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-20T14:45:27.171207145Z INFO: Waiting for worker host service log collection to complete ...
2026-05-20T14:45:27.180524427Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-20T14:45:27.192272736Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-20T14:45:27.274484461Z INFO: Waiting for node performance related collection to complete ...
2026-05-20T14:45:27.708438542Z No resources found in openshift-etcd namespace.
2026-05-20T14:45:27.740068800Z No resources found
2026-05-20T14:45:27.900007942Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-20T14:45:28.307307015Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-20T14:45:28.349839371Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-20T14:45:28.379701049Z INFO: "metallb-operator" not detected. Skipping.
2026-05-20T14:45:28.448171498Z Wrote inspect data to must-gather.
2026-05-20T14:45:28.448212053Z error: inspection completed with the errors occurred while gathering data:
2026-05-20T14:45:28.448212053Z     namespaces "assisted-installer" not found
2026-05-20T14:45:28.549470280Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-20T14:45:28.625914911Z Wrote inspect data to must-gather.
2026-05-20T14:45:28.657386143Z INFO: Worker host service log collection to complete.
2026-05-20T14:45:28.664001046Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-bqvj7
2026-05-20T14:45:28.664056591Z insights-runtime-extractor-7cl6b
2026-05-20T14:45:28.664079702Z insights-runtime-extractor-7gmdx
2026-05-20T14:45:28.664114812Z insights-runtime-extractor-j9rp9
2026-05-20T14:45:28.723936918Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-20T14:45:28.724024957Z INFO: on-disk MachineConfig config collection complete.
2026-05-20T14:45:28.775471436Z error: the server doesn't have a resource type "performanceprofile"
2026-05-20T14:45:29.003763659Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T14:45:29.023662755Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-20T14:45:29.041997484Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-20T14:45:29.345870569Z No resources found
2026-05-20T14:45:29.413555143Z ERROR: No running kube-apiserver pods found
2026-05-20T14:45:29.449877620Z Wrote inspect data to must-gather.
2026-05-20T14:45:29.578981585Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:29.603133946Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-20T14:45:29.603133946Z See 'oc rsync -h' for help and examples
2026-05-20T14:45:29.631440599Z INFO: 'previous-logs' folder not found on ip-10-0-136-116.ec2.internal, skipping...
2026-05-20T14:45:29.686645274Z Wrote inspect data to must-gather.
2026-05-20T14:45:29.686960193Z tar: Removing leading `/' from member names
2026-05-20T14:45:29.695421845Z INFO: HAProxy config collection complete.
2026-05-20T14:45:29.882772854Z Wrote inspect data to must-gather.
2026-05-20T14:45:29.893477097Z Wrote inspect data to must-gather.
2026-05-20T14:45:29.988158739Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-20T14:45:30.002760425Z Wrote inspect data to must-gather.
2026-05-20T14:45:30.171381878Z Gathering data for ns/kuadrant-system...
2026-05-20T14:45:30.181488659Z error: the server doesn't have a resource type "machineconfigs"
2026-05-20T14:45:30.189578219Z INFO: 'previous-logs' folder not found on ip-10-0-137-228.ec2.internal, skipping...
2026-05-20T14:45:30.190595067Z INFO: Getting rules from prometheus-k8s-0
2026-05-20T14:45:30.274420185Z Wrote inspect data to must-gather.
2026-05-20T14:45:30.524724515Z Wrote inspect data to must-gather.
2026-05-20T14:45:30.542380737Z INFO: 'previous-logs' folder not found on ip-10-0-142-178.ec2.internal, skipping...
2026-05-20T14:45:30.542491337Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-20T14:45:30.542573267Z INFO: Machine Config Daemon termination log collection complete.
2026-05-20T14:45:30.634194684Z Wrote inspect data to must-gather.
2026-05-20T14:45:30.739908857Z INFO: Getting status/config from prometheus-k8s-0
2026-05-20T14:45:30.808024496Z Wrote inspect data to must-gather.
2026-05-20T14:45:31.003431141Z Wrote inspect data to must-gather.
2026-05-20T14:45:31.011411899Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-20T14:45:31.239558349Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-20T14:45:31.293057252Z Wrote inspect data to must-gather.
2026-05-20T14:45:31.370456714Z INFO: INTERCONNECT MODE
2026-05-20T14:45:31.370578885Z INFO: Gathering ovn-kubernetes DBs
2026-05-20T14:45:31.516888097Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-20T14:45:31.614264961Z Wrote inspect data to must-gather.
2026-05-20T14:45:31.624651588Z INFO: Gathering OVN_Northbound from ovnkube-node-9nrmg...
2026-05-20T14:45:31.624932157Z INFO: Gathering OVN_Northbound from ovnkube-node-cnh5s...
2026-05-20T14:45:31.625194087Z INFO: Gathering OVN_Northbound from ovnkube-node-dks6t...
2026-05-20T14:45:31.625463139Z INFO: Gathering OVN_Southbound from ovnkube-node-9nrmg...
2026-05-20T14:45:31.625923925Z INFO: Gathering OVN_Southbound from ovnkube-node-cnh5s...
2026-05-20T14:45:31.626210190Z INFO: Gathering OVN_Southbound from ovnkube-node-dks6t...
2026-05-20T14:45:31.644958140Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-20T14:45:31.938864088Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-20T14:45:31.992898172Z tar: Removing leading `/' from member names
2026-05-20T14:45:32.014899377Z tar: Removing leading `/' from member names
2026-05-20T14:45:32.024589767Z tar: Removing leading `/' from member names
2026-05-20T14:45:32.072432951Z tar: Removing leading `/' from member names
2026-05-20T14:45:32.085382094Z tar: Removing leading `/' from member names
2026-05-20T14:45:32.085597317Z tar: Removing leading `/' from member names
2026-05-20T14:45:32.145803949Z Wrote inspect data to must-gather.
2026-05-20T14:45:32.353436911Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-20T14:45:32.379647039Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T14:45:32.578238977Z Wrote inspect data to must-gather.
2026-05-20T14:45:32.686661761Z Gathering data for ns/openshift-monitoring...
2026-05-20T14:45:32.948180553Z INFO: Getting status from alertmanager-main-0
2026-05-20T14:45:32.956378326Z INFO: Waiting for network log collection to complete ...
2026-05-20T14:45:32.956494690Z INFO: Waiting for ovnk database copies to complete ...
2026-05-20T14:45:32.956970199Z INFO: Copying ovnk databases complete.
2026-05-20T14:45:32.958667118Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-05-20T14:45:32.961052780Z ovnk_database_store/
2026-05-20T14:45:32.961098747Z ovnk_database_store/ovnkube-node-9nrmg_sbdb
2026-05-20T14:45:33.010183653Z Wrote inspect data to must-gather.
2026-05-20T14:45:33.017162113Z ovnk_database_store/ovnkube-node-cnh5s_sbdb
2026-05-20T14:45:33.047818275Z ovnk_database_store/ovnkube-node-cnh5s_nbdb
2026-05-20T14:45:33.061543433Z ovnk_database_store/ovnkube-node-9nrmg_nbdb
2026-05-20T14:45:33.074481463Z ovnk_database_store/ovnkube-node-dks6t_sbdb
2026-05-20T14:45:33.104056414Z ovnk_database_store/ovnkube-node-dks6t_nbdb
2026-05-20T14:45:33.118392978Z INFO: Network log collection complete.
2026-05-20T14:45:33.324683420Z Wrote inspect data to must-gather.
2026-05-20T14:45:33.598814608Z Wrote inspect data to must-gather.
2026-05-20T14:45:33.867626782Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-20T14:45:33.872523437Z INFO: Fallback to identify the container image from release info
2026-05-20T14:45:34.519318024Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-20T14:45:34.595827269Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:34.658187404Z daemonset.apps/perf-node-gather-daemonset created
2026-05-20T14:45:34.801715000Z Waiting for performance profile collector pods to become ready: 1
2026-05-20T14:45:35.066808789Z Gathering data for ns/openshift-network-console...
2026-05-20T14:45:35.667730191Z Gathering data for ns/openshift-console-operator...
2026-05-20T14:45:35.925199702Z Waiting for performance profile collector pods to become ready: 2
2026-05-20T14:45:36.155394844Z Gathering data for ns/openshift-console...
2026-05-20T14:45:36.713971950Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-20T14:45:37.034150715Z W0520 14:45:37.034090     293 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-20T14:45:37.068296541Z Waiting for performance profile collector pods to become ready: 3
2026-05-20T14:45:37.154407738Z Wrote inspect data to must-gather.
2026-05-20T14:45:37.185894205Z Gathering data for ns/openshift-dns-operator...
2026-05-20T14:45:37.543850841Z Gathering data for ns/openshift-dns...
2026-05-20T14:45:38.212549119Z Waiting for performance profile collector pods to become ready: 4
2026-05-20T14:45:38.235247264Z Gathering data for ns/openshift-image-registry...
2026-05-20T14:45:38.761613096Z Gathering data for ns/openshift-ingress-operator...
2026-05-20T14:45:39.164550692Z Gathering data for ns/openshift-ingress...
2026-05-20T14:45:39.329679250Z Waiting for performance profile collector pods to become ready: 5
2026-05-20T14:45:39.601441645Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:40.035570188Z Gathering data for ns/openshift-ingress-canary...
2026-05-20T14:45:40.458939646Z Waiting for performance profile collector pods to become ready: 6
2026-05-20T14:45:40.639639734Z Gathering data for ns/openshift-insights...
2026-05-20T14:45:41.572724408Z Waiting for performance profile collector pods to become ready: 7
2026-05-20T14:45:42.694326166Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-20T14:45:42.816245416Z Gathering data for ns/opendatahub...
2026-05-20T14:45:42.941432014Z Collecting performance related data for node ip-10-0-137-228.ec2.internal
2026-05-20T14:45:42.949294117Z Collecting performance related data for node ip-10-0-142-178.ec2.internal
2026-05-20T14:45:42.956858188Z Collecting performance related data for node ip-10-0-136-116.ec2.internal
2026-05-20T14:45:44.606673170Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:44.704424013Z Gathering data for ns/openshift-lws-operator...
2026-05-20T14:45:46.105325959Z Gathering data for ns/openshift-config...
2026-05-20T14:45:46.505354685Z Gathering data for ns/openshift-config-managed...
2026-05-20T14:45:47.033254399Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-20T14:45:47.451718337Z Gathering data for ns/openshift-kube-apiserver...
2026-05-20T14:45:47.830190609Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-20T14:45:48.205051931Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-20T14:45:48.585269820Z Gathering data for ns/openshift-kube-scheduler...
2026-05-20T14:45:48.914854767Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-20T14:45:49.294687822Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-20T14:45:49.612829358Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:49.688630163Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-20T14:45:50.120406940Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-20T14:45:50.735098105Z Gathering data for ns/openshift-multus...
2026-05-20T14:45:51.385598156Z Collecting kubelet logs for node ip-10-0-137-228.ec2.internal
2026-05-20T14:45:51.387393922Z Collecting kubelet logs for node ip-10-0-142-178.ec2.internal
2026-05-20T14:45:51.389124926Z Collecting kubelet logs for node ip-10-0-136-116.ec2.internal
2026-05-20T14:45:51.748262224Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-20T14:45:51.753166164Z INFO: Node performance data collection complete.
2026-05-20T14:45:52.365066534Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-20T14:45:54.020610376Z Gathering data for ns/openshift-host-network...
2026-05-20T14:45:54.391243356Z Gathering data for ns/openshift-network-diagnostics...
2026-05-20T14:45:54.617480598Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:55.009908159Z Gathering data for ns/openshift-network-node-identity...
2026-05-20T14:45:55.514900728Z Gathering data for ns/openshift-network-operator...
2026-05-20T14:45:55.963908049Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-20T14:45:56.306556715Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-20T14:45:56.809954340Z Gathering data for ns/openshift-apiserver-operator...
2026-05-20T14:45:57.150237116Z Gathering data for ns/openshift-apiserver...
2026-05-20T14:45:57.634265790Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-20T14:45:57.949970932Z Gathering data for ns/openshift-controller-manager...
2026-05-20T14:45:58.293396008Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-20T14:45:59.004450447Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-20T14:45:59.334539264Z Gathering data for ns/openshift-service-ca-operator...
2026-05-20T14:45:59.623228330Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-20T14:45:59.696782466Z Gathering data for ns/openshift-service-ca...
2026-05-20T14:46:00.139840254Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-20T14:46:01.236674843Z Wrote inspect data to must-gather.
2026-05-20T14:46:01.236727604Z error: inspection completed with the errors occurred while gathering data:
2026-05-20T14:46:01.236727604Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-20T14:46:01.366714236Z error: the server doesn't have a resource type "clusters"
2026-05-20T14:46:01.715573746Z Caches written to disk