2026-04-23T19:16:56.085247764Z [disk usage checker] Started
2026-04-23T19:16:56.088645362Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:16:56.329396687Z Gathering data for ns/openshift-cluster-version...
2026-04-23T19:16:56.478044197Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T19:16:56.597257452Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T19:16:56.706476031Z Gathering data for ns/default...
2026-04-23T19:16:57.129252296Z Gathering data for ns/openshift...
2026-04-23T19:16:57.633183028Z Gathering data for ns/kube-system...
2026-04-23T19:16:58.438739667Z Gathering data for ns/openshift-etcd...
2026-04-23T19:16:58.706297044Z Waiting on subprocesses to finish execution.
2026-04-23T19:16:58.717078732Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-23T19:16:58.717468786Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-23T19:16:58.718624179Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-23T19:16:58.727788793Z INFO: Gathering HAProxy config files
2026-04-23T19:16:58.734449345Z INFO: Waiting for node performance related collection to complete ...
2026-04-23T19:16:58.741836003Z INFO: Collecting host service logs for crio
2026-04-23T19:16:58.742263257Z INFO: Collecting host service logs for kubelet
2026-04-23T19:16:58.742606901Z INFO: Collecting host service logs for rpm-ostreed
2026-04-23T19:16:58.742944902Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-23T19:16:58.743301847Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-23T19:16:58.743641746Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-23T19:16:58.743976582Z INFO: Collecting host service logs for NetworkManager
2026-04-23T19:16:58.744325214Z INFO: Collecting host service logs for openvswitch
2026-04-23T19:16:58.744655424Z INFO: Collecting host service logs for ovs-configuration
2026-04-23T19:16:58.745028232Z INFO: Collecting host service logs for ovsdb-server
2026-04-23T19:16:58.745374471Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-23T19:16:58.745775849Z INFO: Waiting for worker host service log collection to complete ...
2026-04-23T19:16:59.067937788Z No resources found
2026-04-23T19:16:59.147418731Z Wrote inspect data to must-gather.
2026-04-23T19:16:59.147508223Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T19:16:59.147508223Z     namespaces "assisted-installer" not found
2026-04-23T19:16:59.216438542Z error: the server doesn't have a resource type "performanceprofile"
2026-04-23T19:16:59.306808230Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-6gf74
2026-04-23T19:16:59.306848335Z insights-runtime-extractor-cl2hj
2026-04-23T19:16:59.306867812Z insights-runtime-extractor-dpxlz
2026-04-23T19:16:59.306923697Z insights-runtime-extractor-wsvkb
2026-04-23T19:16:59.606614010Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-23T19:16:59.606707268Z INFO: on-disk MachineConfig config collection complete.
2026-04-23T19:16:59.694151447Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-23T19:16:59.815955658Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-23T19:16:59.815955658Z See 'oc rsync -h' for help and examples
2026-04-23T19:16:59.883772993Z Wrote inspect data to must-gather.
2026-04-23T19:17:00.182956412Z No resources found in openshift-etcd namespace.
2026-04-23T19:17:00.660962122Z INFO: "metallb-operator" not detected. Skipping.
2026-04-23T19:17:00.779325990Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-23T19:17:00.829253358Z INFO: Worker host service log collection to complete.
2026-04-23T19:17:00.976372550Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-23T19:17:01.014283339Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-23T19:17:01.028489640Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-23T19:17:01.118194210Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-23T19:17:01.155840580Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:17:01.198240057Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T19:17:01.373725986Z ERROR: No running kube-apiserver pods found
2026-04-23T19:17:01.442350058Z No resources found
2026-04-23T19:17:01.491064047Z Gathering data for ns/kuadrant-system...
2026-04-23T19:17:01.493936759Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-23T19:17:01.495068661Z Wrote inspect data to must-gather.
2026-04-23T19:17:01.507926659Z error: the server doesn't have a resource type "machineconfigs"
2026-04-23T19:17:01.534596922Z Wrote inspect data to must-gather.
2026-04-23T19:17:01.580429034Z Wrote inspect data to must-gather.
2026-04-23T19:17:01.586388142Z tar: Removing leading `/' from member names
2026-04-23T19:17:01.594864879Z INFO: HAProxy config collection complete.
2026-04-23T19:17:01.678295071Z Wrote inspect data to must-gather.
2026-04-23T19:17:01.691178328Z Wrote inspect data to must-gather.
2026-04-23T19:17:01.807781472Z INFO: 'previous-logs' folder not found on ip-10-0-137-68.ec2.internal, skipping...
2026-04-23T19:17:01.908364451Z Wrote inspect data to must-gather.
2026-04-23T19:17:02.117579356Z Wrote inspect data to must-gather.
2026-04-23T19:17:02.174269976Z INFO: Getting rules from prometheus-k8s-0
2026-04-23T19:17:02.216816013Z INFO: 'previous-logs' folder not found on ip-10-0-138-68.ec2.internal, skipping...
2026-04-23T19:17:02.228687616Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T19:17:02.278621317Z Wrote inspect data to must-gather.
2026-04-23T19:17:02.471342433Z Wrote inspect data to must-gather.
2026-04-23T19:17:02.535505846Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-23T19:17:02.611614331Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T19:17:02.644179451Z INFO: 'previous-logs' folder not found on ip-10-0-143-218.ec2.internal, skipping...
2026-04-23T19:17:02.644226318Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-23T19:17:02.644250139Z INFO: Machine Config Daemon termination log collection complete.
2026-04-23T19:17:02.744599374Z INFO: Getting status/config from prometheus-k8s-0
2026-04-23T19:17:02.894732938Z Gathering data for ns/openshift-monitoring...
2026-04-23T19:17:02.936264498Z Wrote inspect data to must-gather.
2026-04-23T19:17:02.952586454Z Wrote inspect data to must-gather.
2026-04-23T19:17:03.244243429Z Wrote inspect data to must-gather.
2026-04-23T19:17:03.296854736Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-23T19:17:03.326787720Z INFO: INTERCONNECT MODE
2026-04-23T19:17:03.326879775Z INFO: Gathering ovn-kubernetes DBs
2026-04-23T19:17:03.545742755Z INFO: Gathering OVN_Northbound from ovnkube-node-b27s8...
2026-04-23T19:17:03.546012590Z INFO: Gathering OVN_Northbound from ovnkube-node-p7h4b...
2026-04-23T19:17:03.546279664Z INFO: Gathering OVN_Northbound from ovnkube-node-swcqx...
2026-04-23T19:17:03.546714008Z INFO: Gathering OVN_Southbound from ovnkube-node-b27s8...
2026-04-23T19:17:03.547038995Z INFO: Gathering OVN_Southbound from ovnkube-node-p7h4b...
2026-04-23T19:17:03.547353293Z INFO: Gathering OVN_Southbound from ovnkube-node-swcqx...
2026-04-23T19:17:03.569227919Z Wrote inspect data to must-gather.
2026-04-23T19:17:03.590504383Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-23T19:17:03.848074756Z tar: Removing leading `/' from member names
2026-04-23T19:17:03.857185893Z tar: Removing leading `/' from member names
2026-04-23T19:17:03.899711684Z tar: Removing leading `/' from member names
2026-04-23T19:17:03.956604523Z tar: Removing leading `/' from member names
2026-04-23T19:17:04.011117349Z tar: Removing leading `/' from member names
2026-04-23T19:17:04.014668649Z tar: Removing leading `/' from member names
2026-04-23T19:17:04.092132480Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-23T19:17:04.107583109Z Wrote inspect data to must-gather.
2026-04-23T19:17:04.397868717Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-23T19:17:04.467669545Z Wrote inspect data to must-gather.
2026-04-23T19:17:04.682365379Z INFO: Waiting for network log collection to complete ...
2026-04-23T19:17:04.682412821Z INFO: Waiting for ovnk database copies to complete ...
2026-04-23T19:17:04.683015076Z INFO: Copying ovnk databases complete.
2026-04-23T19:17:04.685358123Z 5.2M	must-gather/network_logs/ovnk_database_store
2026-04-23T19:17:04.688387440Z ovnk_database_store/
2026-04-23T19:17:04.688387440Z ovnk_database_store/ovnkube-node-b27s8_nbdb
2026-04-23T19:17:04.703228499Z ovnk_database_store/ovnkube-node-p7h4b_nbdb
2026-04-23T19:17:04.715064238Z ovnk_database_store/ovnkube-node-swcqx_nbdb
2026-04-23T19:17:04.729003185Z ovnk_database_store/ovnkube-node-swcqx_sbdb
2026-04-23T19:17:04.760631296Z ovnk_database_store/ovnkube-node-b27s8_sbdb
2026-04-23T19:17:04.769707147Z Wrote inspect data to must-gather.
2026-04-23T19:17:04.791999972Z ovnk_database_store/ovnkube-node-p7h4b_sbdb
2026-04-23T19:17:04.829678025Z INFO: Network log collection complete.
2026-04-23T19:17:04.903828182Z INFO: Getting status from alertmanager-main-0
2026-04-23T19:17:05.075241862Z Wrote inspect data to must-gather.
2026-04-23T19:17:05.391537969Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-23T19:17:05.394655996Z INFO: Fallback to identify the container image from release info
2026-04-23T19:17:05.508262933Z Gathering data for ns/openshift-network-console...
2026-04-23T19:17:05.979293224Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-23T19:17:06.048745641Z Gathering data for ns/openshift-console-operator...
2026-04-23T19:17:06.142854914Z daemonset.apps/perf-node-gather-daemonset created
2026-04-23T19:17:06.169514022Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:17:06.293453343Z Waiting for performance profile collector pods to become ready: 1
2026-04-23T19:17:06.303570822Z Wrote inspect data to must-gather.
2026-04-23T19:17:06.518302788Z Gathering data for ns/openshift-console...
2026-04-23T19:17:07.032059682Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-23T19:17:07.438332720Z Waiting for performance profile collector pods to become ready: 2
2026-04-23T19:17:07.460055513Z Gathering data for ns/openshift-dns-operator...
2026-04-23T19:17:07.825314018Z Gathering data for ns/openshift-dns...
2026-04-23T19:17:08.573943062Z Gathering data for ns/openshift-image-registry...
2026-04-23T19:17:08.580834330Z Waiting for performance profile collector pods to become ready: 3
2026-04-23T19:17:09.119464277Z Gathering data for ns/openshift-ingress-operator...
2026-04-23T19:17:09.539285180Z Gathering data for ns/openshift-ingress...
2026-04-23T19:17:09.703489559Z Waiting for performance profile collector pods to become ready: 4
2026-04-23T19:17:10.207568202Z Gathering data for ns/openshift-ingress-canary...
2026-04-23T19:17:10.698227830Z Gathering data for ns/openshift-insights...
2026-04-23T19:17:10.830809564Z Waiting for performance profile collector pods to become ready: 5
2026-04-23T19:17:11.177243736Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:17:11.959275398Z Waiting for performance profile collector pods to become ready: 6
2026-04-23T19:17:13.104162365Z Waiting for performance profile collector pods to become ready: 7
2026-04-23T19:17:13.579663955Z Gathering data for ns/openshift-lws-operator...
2026-04-23T19:17:14.176055791Z Gathering data for ns/kserve...
2026-04-23T19:17:14.240202718Z Waiting for performance profile collector pods to become ready: 8
2026-04-23T19:17:15.375665072Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-23T19:17:15.647215268Z Collecting performance related data for node ip-10-0-138-68.ec2.internal
2026-04-23T19:17:15.654412835Z Collecting performance related data for node ip-10-0-137-68.ec2.internal
2026-04-23T19:17:15.662369036Z Collecting performance related data for node ip-10-0-143-218.ec2.internal
2026-04-23T19:17:16.074888421Z Gathering data for ns/openshift-config...
2026-04-23T19:17:16.182450758Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:17:16.470667791Z Gathering data for ns/openshift-config-managed...
2026-04-23T19:17:17.113192118Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-23T19:17:17.528013402Z Gathering data for ns/openshift-kube-apiserver...
2026-04-23T19:17:17.908572015Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-23T19:17:18.323162882Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-23T19:17:18.731650727Z Gathering data for ns/openshift-kube-scheduler...
2026-04-23T19:17:19.131217786Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-23T19:17:19.513541800Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-23T19:17:19.925525117Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-23T19:17:20.324663224Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-23T19:17:21.027724609Z Gathering data for ns/openshift-multus...
2026-04-23T19:17:21.187161657Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:17:22.698393011Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-23T19:17:23.883012344Z Collecting kubelet logs for node ip-10-0-138-68.ec2.internal
2026-04-23T19:17:23.884862056Z Collecting kubelet logs for node ip-10-0-137-68.ec2.internal
2026-04-23T19:17:23.886577516Z Collecting kubelet logs for node ip-10-0-143-218.ec2.internal
2026-04-23T19:17:24.315582412Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-23T19:17:24.321651418Z INFO: Node performance data collection complete.
2026-04-23T19:17:24.515112307Z Gathering data for ns/openshift-host-network...
2026-04-23T19:17:24.908046256Z Gathering data for ns/openshift-network-diagnostics...
2026-04-23T19:17:25.518222295Z Gathering data for ns/openshift-network-node-identity...
2026-04-23T19:17:26.027648229Z Gathering data for ns/openshift-network-operator...
2026-04-23T19:17:26.192731823Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:17:26.481812996Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-23T19:17:26.828461389Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-23T19:17:27.360203964Z Gathering data for ns/openshift-apiserver-operator...
2026-04-23T19:17:27.733864519Z Gathering data for ns/openshift-apiserver...
2026-04-23T19:17:28.213399536Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-23T19:17:28.532518968Z Gathering data for ns/openshift-controller-manager...
2026-04-23T19:17:28.866433537Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-23T19:17:29.587804532Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-23T19:17:29.976597926Z Gathering data for ns/openshift-service-ca-operator...
2026-04-23T19:17:30.345854609Z Gathering data for ns/openshift-service-ca...
2026-04-23T19:17:30.771452054Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-23T19:17:31.199215114Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-23T19:17:31.767637688Z Wrote inspect data to must-gather.
2026-04-23T19:17:31.767680984Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T19:17:31.767680984Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-23T19:17:31.913585186Z error: the server doesn't have a resource type "clusters"
2026-04-23T19:17:32.262039846Z Caches written to disk