2026-05-28T09:59:44.718443210Z [disk usage checker] Started
2026-05-28T09:59:44.722715665Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T09:59:44.929419504Z Gathering data for ns/openshift-cluster-version...
2026-05-28T09:59:45.056323623Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T09:59:45.163727437Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T09:59:45.244956639Z Gathering data for ns/default...
2026-05-28T09:59:45.538770161Z Gathering data for ns/openshift...
2026-05-28T09:59:45.938106639Z Gathering data for ns/kube-system...
2026-05-28T09:59:46.611834160Z Gathering data for ns/openshift-etcd...
2026-05-28T09:59:46.903643167Z Wrote inspect data to must-gather.
2026-05-28T09:59:46.903676036Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T09:59:46.903676036Z     namespaces "assisted-installer" not found
2026-05-28T09:59:47.051102278Z Waiting on subprocesses to finish execution.
2026-05-28T09:59:47.060052354Z INFO: Collecting host service logs for crio
2026-05-28T09:59:47.060339489Z INFO: Collecting host service logs for kubelet
2026-05-28T09:59:47.060458814Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T09:59:47.060642996Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T09:59:47.061147490Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T09:59:47.061492828Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T09:59:47.061858769Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T09:59:47.062168337Z INFO: Collecting host service logs for NetworkManager
2026-05-28T09:59:47.062538093Z INFO: Collecting host service logs for openvswitch
2026-05-28T09:59:47.063609242Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T09:59:47.063964189Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T09:59:47.064276283Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T09:59:47.064884198Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T09:59:47.074895534Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T09:59:47.085160292Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T09:59:47.091517567Z INFO: Gathering HAProxy config files
2026-05-28T09:59:47.293840756Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T09:59:47.549210370Z No resources found
2026-05-28T09:59:47.588456414Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T09:59:47.640847137Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T09:59:47.737589894Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T09:59:48.173075744Z ERROR: No running kube-apiserver pods found
2026-05-28T09:59:48.178069080Z No resources found in openshift-etcd namespace.
2026-05-28T09:59:48.288862738Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T09:59:48.288862738Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T09:59:48.304559847Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T09:59:48.422592169Z INFO: Worker host service log collection to complete.
2026-05-28T09:59:48.467437655Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-b7lgj
2026-05-28T09:59:48.467500550Z insights-runtime-extractor-lqlpn
2026-05-28T09:59:48.467522104Z insights-runtime-extractor-tw2ql
2026-05-28T09:59:48.467543091Z insights-runtime-extractor-vln56
2026-05-28T09:59:48.500578514Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T09:59:48.508611697Z Wrote inspect data to must-gather.
2026-05-28T09:59:48.523096137Z Wrote inspect data to must-gather.
2026-05-28T09:59:48.562321652Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T09:59:48.593275000Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T09:59:48.708097455Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T09:59:48.708097455Z See 'oc rsync -h' for help and examples
2026-05-28T09:59:48.825825762Z INFO: 'previous-logs' folder not found on ip-10-0-128-101.ec2.internal, skipping...
2026-05-28T09:59:48.875899225Z Wrote inspect data to must-gather.
2026-05-28T09:59:48.889126104Z Gathering data for ns/openshift-monitoring...
2026-05-28T09:59:49.166159906Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T09:59:49.186958072Z tar: Removing leading `/' from member names
2026-05-28T09:59:49.193720688Z INFO: HAProxy config collection complete.
2026-05-28T09:59:49.198315489Z No resources found
2026-05-28T09:59:49.295548231Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T09:59:49.343647297Z Wrote inspect data to must-gather.
2026-05-28T09:59:49.353999217Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T09:59:49.432133925Z Wrote inspect data to must-gather.
2026-05-28T09:59:49.438300594Z Wrote inspect data to must-gather.
2026-05-28T09:59:49.463289066Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T09:59:49.506014222Z Wrote inspect data to must-gather.
2026-05-28T09:59:49.582670733Z INFO: 'previous-logs' folder not found on ip-10-0-129-139.ec2.internal, skipping...
2026-05-28T09:59:49.709259373Z Wrote inspect data to must-gather.
2026-05-28T09:59:49.713640750Z Wrote inspect data to must-gather.
2026-05-28T09:59:49.732396223Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T09:59:49.843291919Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T09:59:49.847876581Z INFO: 'previous-logs' folder not found on ip-10-0-139-204.ec2.internal, skipping...
2026-05-28T09:59:49.847918023Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T09:59:49.847950094Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T09:59:49.937390785Z Wrote inspect data to must-gather.
2026-05-28T09:59:49.938084299Z Wrote inspect data to must-gather.
2026-05-28T09:59:50.081882540Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T09:59:50.164457444Z Wrote inspect data to must-gather.
2026-05-28T09:59:50.215227153Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T09:59:50.293363835Z Wrote inspect data to must-gather.
2026-05-28T09:59:50.459166156Z INFO: INTERCONNECT MODE
2026-05-28T09:59:50.459203718Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T09:59:50.515849870Z Wrote inspect data to must-gather.
2026-05-28T09:59:50.572668646Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T09:59:50.601111173Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T09:59:50.614864311Z INFO: Gathering OVN_Northbound from ovnkube-node-dd84z...
2026-05-28T09:59:50.615209343Z INFO: Gathering OVN_Northbound from ovnkube-node-hxhf6...
2026-05-28T09:59:50.615521381Z INFO: Gathering OVN_Northbound from ovnkube-node-xvwzm...
2026-05-28T09:59:50.616239468Z INFO: Gathering OVN_Southbound from ovnkube-node-dd84z...
2026-05-28T09:59:50.616550482Z INFO: Gathering OVN_Southbound from ovnkube-node-hxhf6...
2026-05-28T09:59:50.616868226Z INFO: Gathering OVN_Southbound from ovnkube-node-xvwzm...
2026-05-28T09:59:50.813131124Z Wrote inspect data to must-gather.
2026-05-28T09:59:51.041397948Z tar: Removing leading `/' from member names
2026-05-28T09:59:51.045521073Z tar: Removing leading `/' from member names
2026-05-28T09:59:51.090306325Z tar: Removing leading `/' from member names
2026-05-28T09:59:51.094512747Z tar: Removing leading `/' from member names
2026-05-28T09:59:51.096818105Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T09:59:51.097968737Z tar: Removing leading `/' from member names
2026-05-28T09:59:51.100130701Z tar: Removing leading `/' from member names
2026-05-28T09:59:51.101109954Z Wrote inspect data to must-gather.
2026-05-28T09:59:51.359901929Z W0528 09:59:51.359840    1309 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-28T09:59:51.367044459Z Wrote inspect data to must-gather.
2026-05-28T09:59:51.377520763Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T09:59:51.422001785Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T09:59:51.482046564Z INFO: Waiting for network log collection to complete ...
2026-05-28T09:59:51.482082611Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T09:59:51.482444721Z INFO: Copying ovnk databases complete.
2026-05-28T09:59:51.484177052Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-05-28T09:59:51.486653224Z ovnk_database_store/
2026-05-28T09:59:51.486653224Z ovnk_database_store/ovnkube-node-xvwzm_nbdb
2026-05-28T09:59:51.525066918Z ovnk_database_store/ovnkube-node-xvwzm_sbdb
2026-05-28T09:59:51.548673903Z ovnk_database_store/ovnkube-node-dd84z_nbdb
2026-05-28T09:59:51.556966055Z ovnk_database_store/ovnkube-node-hxhf6_sbdb
2026-05-28T09:59:51.579101787Z ovnk_database_store/ovnkube-node-dd84z_sbdb
2026-05-28T09:59:51.598817843Z ovnk_database_store/ovnkube-node-hxhf6_nbdb
2026-05-28T09:59:51.608434449Z Gathering data for ns/openshift-network-console...
2026-05-28T09:59:51.609283736Z INFO: Network log collection complete.
2026-05-28T09:59:51.660841452Z Wrote inspect data to must-gather.
2026-05-28T09:59:51.712216784Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T09:59:51.935981355Z Wrote inspect data to must-gather.
2026-05-28T09:59:52.139185882Z Gathering data for ns/openshift-console-operator...
2026-05-28T09:59:52.142874451Z INFO: Getting status from alertmanager-main-0
2026-05-28T09:59:52.207006418Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T09:59:52.210768490Z INFO: Fallback to identify the container image from release info
2026-05-28T09:59:52.590985949Z Gathering data for ns/openshift-console...
2026-05-28T09:59:52.790925482Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T09:59:52.916072379Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T09:59:53.051780653Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T09:59:53.082354048Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T09:59:53.500330351Z Gathering data for ns/openshift-dns-operator...
2026-05-28T09:59:53.833852692Z Gathering data for ns/openshift-dns...
2026-05-28T09:59:54.178937780Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T09:59:54.435240202Z Gathering data for ns/openshift-image-registry...
2026-05-28T09:59:54.738333547Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T09:59:54.943152756Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T09:59:55.277152906Z Gathering data for ns/openshift-ingress...
2026-05-28T09:59:55.301490241Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T09:59:55.614129279Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T09:59:55.989957662Z Gathering data for ns/openshift-insights...
2026-05-28T09:59:56.419092789Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T09:59:57.537682674Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T09:59:58.001927470Z Gathering data for ns/openshift-config...
2026-05-28T09:59:58.252730095Z Gathering data for ns/openshift-config-managed...
2026-05-28T09:59:58.664881115Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T09:59:58.668240168Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T09:59:58.960489906Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T09:59:59.199923221Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T09:59:59.483835382Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T09:59:59.743383827Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T09:59:59.785594927Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T09:59:59.791625088Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T10:00:00.038261378Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T10:00:00.366313729Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T10:00:00.672171545Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T10:00:00.904234710Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T10:00:01.003839287Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T10:00:01.236459042Z Collecting performance related data for node ip-10-0-128-101.ec2.internal
2026-05-28T10:00:01.246053976Z Collecting performance related data for node ip-10-0-129-139.ec2.internal
2026-05-28T10:00:01.256469113Z Collecting performance related data for node ip-10-0-139-204.ec2.internal
2026-05-28T10:00:01.605164940Z Gathering data for ns/openshift-multus...
2026-05-28T10:00:03.039741797Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T10:00:04.581106646Z Gathering data for ns/openshift-host-network...
2026-05-28T10:00:04.748548841Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T10:00:04.891308044Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T10:00:05.469008922Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T10:00:05.976166559Z Gathering data for ns/openshift-network-operator...
2026-05-28T10:00:06.367063282Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T10:00:06.635054960Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T10:00:07.149236431Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T10:00:07.489863030Z Gathering data for ns/openshift-apiserver...
2026-05-28T10:00:07.914317135Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T10:00:08.208956205Z Gathering data for ns/openshift-controller-manager...
2026-05-28T10:00:08.504128398Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T10:00:09.081365983Z Collecting kubelet logs for node ip-10-0-128-101.ec2.internal
2026-05-28T10:00:09.083347173Z Collecting kubelet logs for node ip-10-0-129-139.ec2.internal
2026-05-28T10:00:09.085390210Z Collecting kubelet logs for node ip-10-0-139-204.ec2.internal
2026-05-28T10:00:09.218876434Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T10:00:09.455928201Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T10:00:09.461607282Z INFO: Node performance data collection complete.
2026-05-28T10:00:09.544148831Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T10:00:09.753619348Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-28T10:00:09.856662091Z Gathering data for ns/openshift-service-ca...
2026-05-28T10:00:10.220642402Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T10:00:11.126949993Z Wrote inspect data to must-gather.
2026-05-28T10:00:11.127004107Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T10:00:11.127004107Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T10:00:11.256764991Z error: the server doesn't have a resource type "clusters"
2026-05-28T10:00:11.383489589Z Caches written to disk