2026-05-25T01:38:08.881407837Z [disk usage checker] Started
2026-05-25T01:38:08.884913518Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-05-25T01:38:09.104715315Z Gathering data for ns/openshift-cluster-version...
2026-05-25T01:38:09.289418896Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T01:38:09.395541460Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T01:38:09.497630827Z Gathering data for ns/default...
2026-05-25T01:38:09.862104590Z Gathering data for ns/openshift...
2026-05-25T01:38:10.328263427Z Gathering data for ns/kube-system...
2026-05-25T01:38:11.065792688Z Gathering data for ns/openshift-etcd...
2026-05-25T01:38:11.283128811Z Waiting on subprocesses to finish execution.
2026-05-25T01:38:11.287769682Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-25T01:38:11.290982789Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-25T01:38:11.302548281Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-25T01:38:11.304765627Z INFO: Gathering HAProxy config files
2026-05-25T01:38:11.343320053Z INFO: Waiting for node performance related collection to complete ...
2026-05-25T01:38:11.403768181Z INFO: Collecting host service logs for crio
2026-05-25T01:38:11.404080795Z INFO: Collecting host service logs for kubelet
2026-05-25T01:38:11.404405740Z INFO: Collecting host service logs for rpm-ostreed
2026-05-25T01:38:11.404689110Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-25T01:38:11.404962886Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-25T01:38:11.405277127Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-25T01:38:11.405598437Z INFO: Collecting host service logs for NetworkManager
2026-05-25T01:38:11.405870757Z INFO: Collecting host service logs for openvswitch
2026-05-25T01:38:11.406175171Z INFO: Collecting host service logs for ovs-configuration
2026-05-25T01:38:11.406512465Z INFO: Collecting host service logs for ovsdb-server
2026-05-25T01:38:11.406807975Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-25T01:38:11.407192118Z INFO: Waiting for worker host service log collection to complete ...
2026-05-25T01:38:11.855112174Z Wrote inspect data to must-gather.
2026-05-25T01:38:11.855202794Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T01:38:11.855202794Z     namespaces "assisted-installer" not found
2026-05-25T01:38:11.943981809Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-25T01:38:12.173818545Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-25T01:38:12.352616922Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-25T01:38:12.352718835Z INFO: on-disk MachineConfig config collection complete.
2026-05-25T01:38:12.471915957Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-25T01:38:12.509703195Z No resources found in openshift-etcd namespace.
2026-05-25T01:38:12.527314391Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-rmjjk
2026-05-25T01:38:12.527377951Z insights-runtime-extractor-grsvl
2026-05-25T01:38:12.527398783Z insights-runtime-extractor-lc5qg
2026-05-25T01:38:12.527418573Z insights-runtime-extractor-n2vnl
2026-05-25T01:38:12.593100068Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-25T01:38:12.629833980Z INFO: "metallb-operator" not detected. Skipping.
2026-05-25T01:38:12.726948885Z No resources found
2026-05-25T01:38:12.794699008Z Wrote inspect data to must-gather.
2026-05-25T01:38:12.849647665Z INFO: Worker host service log collection to complete.
2026-05-25T01:38:12.891314594Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-25T01:38:12.891314594Z See 'oc rsync -h' for help and examples
2026-05-25T01:38:12.965143965Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-25T01:38:13.176280669Z error: the server doesn't have a resource type "performanceprofile"
2026-05-25T01:38:13.176280669Z ERROR: No running kube-apiserver pods found
2026-05-25T01:38:13.379190514Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-25T01:38:13.464909273Z INFO: 'previous-logs' folder not found on ip-10-0-133-121.ec2.internal, skipping...
2026-05-25T01:38:13.516096747Z No resources found
2026-05-25T01:38:13.764708964Z Wrote inspect data to must-gather.
2026-05-25T01:38:13.808609422Z Wrote inspect data to must-gather.
2026-05-25T01:38:13.841713618Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T01:38:13.909267841Z Wrote inspect data to must-gather.
2026-05-25T01:38:13.914731221Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-05-25T01:38:13.921834539Z Wrote inspect data to must-gather.
2026-05-25T01:38:14.072393727Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-25T01:38:14.169189804Z Gathering data for ns/kuadrant-system...
2026-05-25T01:38:14.181743815Z INFO: Getting rules from prometheus-k8s-0
2026-05-25T01:38:14.193266583Z Wrote inspect data to must-gather.
2026-05-25T01:38:14.202854650Z tar: Removing leading `/' from member names
2026-05-25T01:38:14.213036420Z INFO: HAProxy config collection complete.
2026-05-25T01:38:14.227863823Z INFO: 'previous-logs' folder not found on ip-10-0-134-192.ec2.internal, skipping...
2026-05-25T01:38:14.246362191Z error: the server doesn't have a resource type "machineconfigs"
2026-05-25T01:38:14.569397130Z INFO: 'previous-logs' folder not found on ip-10-0-142-62.ec2.internal, skipping...
2026-05-25T01:38:14.569517011Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-25T01:38:14.569581072Z INFO: Machine Config Daemon termination log collection complete.
2026-05-25T01:38:14.573019171Z Wrote inspect data to must-gather.
2026-05-25T01:38:14.602193502Z Wrote inspect data to must-gather.
2026-05-25T01:38:14.702704974Z INFO: Getting status/config from prometheus-k8s-0
2026-05-25T01:38:14.888102294Z Wrote inspect data to must-gather.
2026-05-25T01:38:14.956501165Z Wrote inspect data to must-gather.
2026-05-25T01:38:15.116539299Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-25T01:38:15.284989142Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T01:38:15.286433106Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-25T01:38:15.345152995Z Wrote inspect data to must-gather.
2026-05-25T01:38:15.423498296Z Wrote inspect data to must-gather.
2026-05-25T01:38:15.543721903Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-25T01:38:15.709289048Z INFO: INTERCONNECT MODE
2026-05-25T01:38:15.709399465Z INFO: Gathering ovn-kubernetes DBs
2026-05-25T01:38:15.752061532Z Wrote inspect data to must-gather.
2026-05-25T01:38:15.830791599Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-25T01:38:15.894064976Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T01:38:15.916228197Z INFO: Gathering OVN_Northbound from ovnkube-node-489fr...
2026-05-25T01:38:15.917442365Z INFO: Gathering OVN_Northbound from ovnkube-node-mmvhx...
2026-05-25T01:38:15.917755048Z INFO: Gathering OVN_Northbound from ovnkube-node-rj8vk...
2026-05-25T01:38:15.918063560Z INFO: Gathering OVN_Southbound from ovnkube-node-489fr...
2026-05-25T01:38:15.918361394Z INFO: Gathering OVN_Southbound from ovnkube-node-mmvhx...
2026-05-25T01:38:15.919682001Z INFO: Gathering OVN_Southbound from ovnkube-node-rj8vk...
2026-05-25T01:38:16.216029024Z tar: Removing leading `/' from member names
2026-05-25T01:38:16.270891800Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-25T01:38:16.286976987Z tar: Removing leading `/' from member names
2026-05-25T01:38:16.291489682Z Wrote inspect data to must-gather.
2026-05-25T01:38:16.300712158Z tar: Removing leading `/' from member names
2026-05-25T01:38:16.302978812Z Gathering data for ns/openshift-monitoring...
2026-05-25T01:38:16.348262200Z tar: Removing leading `/' from member names
2026-05-25T01:38:16.351493076Z tar: Removing leading `/' from member names
2026-05-25T01:38:16.388591280Z tar: Removing leading `/' from member names
2026-05-25T01:38:16.727702787Z Wrote inspect data to must-gather.
2026-05-25T01:38:16.802568069Z INFO: Getting status from alertmanager-main-0
2026-05-25T01:38:17.075047969Z INFO: Waiting for network log collection to complete ...
2026-05-25T01:38:17.075129447Z INFO: Waiting for ovnk database copies to complete ...
2026-05-25T01:38:17.075571193Z INFO: Copying ovnk databases complete.
2026-05-25T01:38:17.077823972Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-05-25T01:38:17.080716961Z ovnk_database_store/
2026-05-25T01:38:17.080716961Z ovnk_database_store/ovnkube-node-mmvhx_sbdb
2026-05-25T01:38:17.106414739Z ovnk_database_store/ovnkube-node-rj8vk_sbdb
2026-05-25T01:38:17.137040492Z ovnk_database_store/ovnkube-node-mmvhx_nbdb
2026-05-25T01:38:17.146559521Z ovnk_database_store/ovnkube-node-489fr_sbdb
2026-05-25T01:38:17.149639221Z Wrote inspect data to must-gather.
2026-05-25T01:38:17.172427852Z ovnk_database_store/ovnkube-node-rj8vk_nbdb
2026-05-25T01:38:17.186680560Z ovnk_database_store/ovnkube-node-489fr_nbdb
2026-05-25T01:38:17.201571117Z INFO: Network log collection complete.
2026-05-25T01:38:17.451144508Z Wrote inspect data to must-gather.
2026-05-25T01:38:17.797230890Z Wrote inspect data to must-gather.
2026-05-25T01:38:18.110972103Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-25T01:38:18.113477188Z INFO: Fallback to identify the container image from release info
2026-05-25T01:38:18.581045227Z Gathering data for ns/openshift-network-console...
2026-05-25T01:38:18.677187525Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-25T01:38:18.815699178Z daemonset.apps/perf-node-gather-daemonset created
2026-05-25T01:38:18.935421394Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-05-25T01:38:18.980938336Z Waiting for performance profile collector pods to become ready: 1
2026-05-25T01:38:19.131501692Z Gathering data for ns/openshift-console-operator...
2026-05-25T01:38:19.645906352Z Gathering data for ns/openshift-console...
2026-05-25T01:38:20.127414794Z Waiting for performance profile collector pods to become ready: 2
2026-05-25T01:38:20.218806528Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-25T01:38:20.651513294Z Gathering data for ns/openshift-dns-operator...
2026-05-25T01:38:20.978374871Z Wrote inspect data to must-gather.
2026-05-25T01:38:21.002806763Z Gathering data for ns/openshift-dns...
2026-05-25T01:38:21.240035960Z Waiting for performance profile collector pods to become ready: 3
2026-05-25T01:38:21.618115565Z Gathering data for ns/openshift-image-registry...
2026-05-25T01:38:22.112276831Z Gathering data for ns/openshift-ingress-operator...
2026-05-25T01:38:22.358029647Z Waiting for performance profile collector pods to become ready: 4
2026-05-25T01:38:22.501976063Z Gathering data for ns/openshift-ingress...
2026-05-25T01:38:23.262444182Z Gathering data for ns/openshift-ingress-canary...
2026-05-25T01:38:23.489473215Z Waiting for performance profile collector pods to become ready: 5
2026-05-25T01:38:23.755957244Z Gathering data for ns/openshift-insights...
2026-05-25T01:38:23.940557807Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-05-25T01:38:24.603995932Z Waiting for performance profile collector pods to become ready: 6
2026-05-25T01:38:25.698911975Z Gathering data for ns/opendatahub...
2026-05-25T01:38:25.714090908Z Waiting for performance profile collector pods to become ready: 7
2026-05-25T01:38:26.832628307Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-25T01:38:27.078066226Z Collecting performance related data for node ip-10-0-134-192.ec2.internal
2026-05-25T01:38:27.085536825Z Collecting performance related data for node ip-10-0-133-121.ec2.internal
2026-05-25T01:38:27.093217783Z Collecting performance related data for node ip-10-0-142-62.ec2.internal
2026-05-25T01:38:27.132683694Z Gathering data for ns/openshift-lws-operator...
2026-05-25T01:38:28.684847091Z Gathering data for ns/openshift-config...
2026-05-25T01:38:28.945976097Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-05-25T01:38:29.083781392Z Gathering data for ns/openshift-config-managed...
2026-05-25T01:38:29.589911599Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-25T01:38:29.935560766Z Gathering data for ns/openshift-kube-apiserver...
2026-05-25T01:38:30.278464309Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-25T01:38:30.630790859Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-25T01:38:30.994966907Z Gathering data for ns/openshift-kube-scheduler...
2026-05-25T01:38:31.320257129Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-25T01:38:31.678025394Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-25T01:38:32.042806835Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-25T01:38:32.406945717Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-25T01:38:32.966680053Z Gathering data for ns/openshift-multus...
2026-05-25T01:38:33.951104515Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-05-25T01:38:34.439428365Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-25T01:38:35.483895432Z Collecting kubelet logs for node ip-10-0-134-192.ec2.internal
2026-05-25T01:38:35.486086712Z Collecting kubelet logs for node ip-10-0-133-121.ec2.internal
2026-05-25T01:38:35.487922298Z Collecting kubelet logs for node ip-10-0-142-62.ec2.internal
2026-05-25T01:38:35.876433022Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-25T01:38:35.881983117Z INFO: Node performance data collection complete.
2026-05-25T01:38:36.021254119Z Gathering data for ns/openshift-host-network...
2026-05-25T01:38:36.405158501Z Gathering data for ns/openshift-network-diagnostics...
2026-05-25T01:38:36.964866924Z Gathering data for ns/openshift-network-node-identity...
2026-05-25T01:38:37.430689199Z Gathering data for ns/openshift-network-operator...
2026-05-25T01:38:37.833373090Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-25T01:38:38.187005572Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-25T01:38:38.661733008Z Gathering data for ns/openshift-apiserver-operator...
2026-05-25T01:38:38.958045992Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-05-25T01:38:38.986509567Z Gathering data for ns/openshift-apiserver...
2026-05-25T01:38:39.445818978Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-25T01:38:39.782447960Z Gathering data for ns/openshift-controller-manager...
2026-05-25T01:38:40.112091695Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-25T01:38:40.797281377Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-25T01:38:41.122630314Z Gathering data for ns/openshift-service-ca-operator...
2026-05-25T01:38:41.462572256Z Gathering data for ns/openshift-service-ca...
2026-05-25T01:38:41.850609574Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-25T01:38:42.841078208Z Wrote inspect data to must-gather.
2026-05-25T01:38:42.841122736Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T01:38:42.841122736Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-25T01:38:42.973985425Z error: the server doesn't have a resource type "clusters"
2026-05-25T01:38:43.135853148Z Caches written to disk