2026-04-22T19:08:16.890828121Z [disk usage checker] Started
2026-04-22T19:08:16.894181119Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-22T19:08:17.093965207Z Gathering data for ns/openshift-cluster-version...
2026-04-22T19:08:17.213197055Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T19:08:17.296310486Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T19:08:17.386292451Z Gathering data for ns/default...
2026-04-22T19:08:17.688349318Z Gathering data for ns/openshift...
2026-04-22T19:08:18.084143638Z Gathering data for ns/kube-system...
2026-04-22T19:08:18.764970033Z Gathering data for ns/openshift-etcd...
2026-04-22T19:08:19.100203066Z Wrote inspect data to must-gather.
2026-04-22T19:08:19.100239980Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T19:08:19.100239980Z     namespaces "assisted-installer" not found
2026-04-22T19:08:19.321731267Z Waiting on subprocesses to finish execution.
2026-04-22T19:08:19.327768128Z INFO: Gathering HAProxy config files
2026-04-22T19:08:19.331257095Z INFO: Collecting host service logs for crio
2026-04-22T19:08:19.331604188Z INFO: Collecting host service logs for kubelet
2026-04-22T19:08:19.331950545Z INFO: Collecting host service logs for rpm-ostreed
2026-04-22T19:08:19.332314541Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-22T19:08:19.332664686Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-22T19:08:19.333299231Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-22T19:08:19.333649816Z INFO: Collecting host service logs for NetworkManager
2026-04-22T19:08:19.333963425Z INFO: Collecting host service logs for openvswitch
2026-04-22T19:08:19.334287457Z INFO: Collecting host service logs for ovs-configuration
2026-04-22T19:08:19.334652475Z INFO: Collecting host service logs for ovsdb-server
2026-04-22T19:08:19.334985706Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-22T19:08:19.335388961Z INFO: Waiting for worker host service log collection to complete ...
2026-04-22T19:08:19.340768403Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-22T19:08:19.350286189Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-22T19:08:19.400547581Z INFO: Waiting for node performance related collection to complete ...
2026-04-22T19:08:19.460862202Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-22T19:08:19.766275490Z INFO: "metallb-operator" not detected. Skipping.
2026-04-22T19:08:19.771100914Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-22T19:08:19.789699061Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-22T19:08:19.897984155Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-22T19:08:20.060956528Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-tnfzj
2026-04-22T19:08:20.060956528Z insights-runtime-extractor-42q2k
2026-04-22T19:08:20.060956528Z insights-runtime-extractor-6m5dg
2026-04-22T19:08:20.060956528Z insights-runtime-extractor-kcplt
2026-04-22T19:08:20.493105626Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-22T19:08:20.493396786Z INFO: Worker host service log collection to complete.
2026-04-22T19:08:20.551893936Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-22T19:08:20.551893936Z See 'oc rsync -h' for help and examples
2026-04-22T19:08:20.557267446Z No resources found
2026-04-22T19:08:20.609285842Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-22T19:08:20.623279091Z error: the server doesn't have a resource type "performanceprofile"
2026-04-22T19:08:20.666365571Z ERROR: No running kube-apiserver pods found
2026-04-22T19:08:20.668412189Z No resources found in openshift-etcd namespace.
2026-04-22T19:08:20.734710530Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-22T19:08:20.734789721Z INFO: on-disk MachineConfig config collection complete.
2026-04-22T19:08:20.745524218Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-22T19:08:20.882756313Z Wrote inspect data to must-gather.
2026-04-22T19:08:21.049889842Z Wrote inspect data to must-gather.
2026-04-22T19:08:21.323089982Z Wrote inspect data to must-gather.
2026-04-22T19:08:21.374276297Z tar: Removing leading `/' from member names
2026-04-22T19:08:21.383568651Z INFO: HAProxy config collection complete.
2026-04-22T19:08:21.411270014Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T19:08:21.557641245Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-22T19:08:21.602518104Z No resources found
2026-04-22T19:08:21.642255482Z Wrote inspect data to must-gather.
2026-04-22T19:08:21.728202612Z Wrote inspect data to must-gather.
2026-04-22T19:08:21.735665353Z Wrote inspect data to must-gather.
2026-04-22T19:08:21.737787213Z INFO: 'previous-logs' folder not found on ip-10-0-128-208.ec2.internal, skipping...
2026-04-22T19:08:21.759545838Z error: the server doesn't have a resource type "machineconfigs"
2026-04-22T19:08:21.773839424Z Wrote inspect data to must-gather.
2026-04-22T19:08:21.781409043Z INFO: Getting rules from prometheus-k8s-0
2026-04-22T19:08:21.867843657Z Gathering data for ns/openshift-monitoring...
2026-04-22T19:08:21.903784857Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-22T19:08:21.982101279Z Wrote inspect data to must-gather.
2026-04-22T19:08:22.006063368Z Wrote inspect data to must-gather.
2026-04-22T19:08:22.010597584Z INFO: 'previous-logs' folder not found on ip-10-0-133-42.ec2.internal, skipping...
2026-04-22T19:08:22.234522044Z INFO: Getting status/config from prometheus-k8s-0
2026-04-22T19:08:22.238846285Z Wrote inspect data to must-gather.
2026-04-22T19:08:22.250575436Z Wrote inspect data to must-gather.
2026-04-22T19:08:22.282772733Z INFO: 'previous-logs' folder not found on ip-10-0-136-85.ec2.internal, skipping...
2026-04-22T19:08:22.282812455Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-22T19:08:22.282826577Z INFO: Machine Config Daemon termination log collection complete.
2026-04-22T19:08:22.395412033Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-22T19:08:22.487456702Z Wrote inspect data to must-gather.
2026-04-22T19:08:22.618060129Z Wrote inspect data to must-gather.
2026-04-22T19:08:22.642292505Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-22T19:08:22.777740931Z INFO: INTERCONNECT MODE
2026-04-22T19:08:22.777772169Z INFO: Gathering ovn-kubernetes DBs
2026-04-22T19:08:22.829778636Z Wrote inspect data to must-gather.
2026-04-22T19:08:22.867400048Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-22T19:08:22.945848545Z INFO: Gathering OVN_Northbound from ovnkube-node-b8wsf...
2026-04-22T19:08:22.946139661Z INFO: Gathering OVN_Northbound from ovnkube-node-vmbf4...
2026-04-22T19:08:22.946406181Z INFO: Gathering OVN_Northbound from ovnkube-node-zf6t2...
2026-04-22T19:08:22.946688896Z INFO: Gathering OVN_Southbound from ovnkube-node-b8wsf...
2026-04-22T19:08:22.947083198Z INFO: Gathering OVN_Southbound from ovnkube-node-vmbf4...
2026-04-22T19:08:22.947413383Z INFO: Gathering OVN_Southbound from ovnkube-node-zf6t2...
2026-04-22T19:08:23.107530614Z Wrote inspect data to must-gather.
2026-04-22T19:08:23.185909399Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-22T19:08:23.262233176Z tar: Removing leading `/' from member names
2026-04-22T19:08:23.272442337Z tar: Removing leading `/' from member names
2026-04-22T19:08:23.272525729Z tar: Removing leading `/' from member names
2026-04-22T19:08:23.276492272Z tar: Removing leading `/' from member names
2026-04-22T19:08:23.283008978Z tar: Removing leading `/' from member names
2026-04-22T19:08:23.338047519Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T19:08:23.342543632Z tar: Removing leading `/' from member names
2026-04-22T19:08:23.411241785Z Wrote inspect data to must-gather.
2026-04-22T19:08:23.473368145Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-22T19:08:23.672484835Z Wrote inspect data to must-gather.
2026-04-22T19:08:23.770524757Z INFO: Waiting for network log collection to complete ...
2026-04-22T19:08:23.770645898Z INFO: Waiting for ovnk database copies to complete ...
2026-04-22T19:08:23.771252154Z INFO: Copying ovnk databases complete.
2026-04-22T19:08:23.773426637Z 3.3M	must-gather/network_logs/ovnk_database_store
2026-04-22T19:08:23.775964810Z ovnk_database_store/
2026-04-22T19:08:23.775964810Z ovnk_database_store/ovnkube-node-zf6t2_nbdb
2026-04-22T19:08:23.787433256Z ovnk_database_store/ovnkube-node-b8wsf_nbdb
2026-04-22T19:08:23.796138639Z ovnk_database_store/ovnkube-node-b8wsf_sbdb
2026-04-22T19:08:23.819141015Z ovnk_database_store/ovnkube-node-vmbf4_sbdb
2026-04-22T19:08:23.839773951Z ovnk_database_store/ovnkube-node-zf6t2_sbdb
2026-04-22T19:08:23.858309071Z ovnk_database_store/ovnkube-node-vmbf4_nbdb
2026-04-22T19:08:23.868174879Z INFO: Network log collection complete.
2026-04-22T19:08:23.896369823Z W0422 19:08:23.896331    1383 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-22T19:08:23.902035541Z INFO: Getting status from alertmanager-main-0
2026-04-22T19:08:23.903715349Z Wrote inspect data to must-gather.
2026-04-22T19:08:23.968564711Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T19:08:24.142918554Z Gathering data for ns/openshift-network-console...
2026-04-22T19:08:24.203585238Z Wrote inspect data to must-gather.
2026-04-22T19:08:24.460804035Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-22T19:08:24.463715331Z INFO: Fallback to identify the container image from release info
2026-04-22T19:08:24.509468464Z Gathering data for ns/openshift-console-operator...
2026-04-22T19:08:24.838781231Z Gathering data for ns/openshift-console...
2026-04-22T19:08:24.969455001Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-22T19:08:25.100541658Z daemonset.apps/perf-node-gather-daemonset created
2026-04-22T19:08:25.237325465Z Waiting for performance profile collector pods to become ready: 1
2026-04-22T19:08:25.269795172Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-22T19:08:25.625555687Z Gathering data for ns/openshift-dns-operator...
2026-04-22T19:08:25.918046210Z Gathering data for ns/openshift-dns...
2026-04-22T19:08:26.368341616Z Waiting for performance profile collector pods to become ready: 2
2026-04-22T19:08:26.461183143Z Gathering data for ns/openshift-image-registry...
2026-04-22T19:08:26.871165239Z Gathering data for ns/openshift-ingress-operator...
2026-04-22T19:08:26.909775826Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-22T19:08:27.202946325Z Gathering data for ns/openshift-ingress...
2026-04-22T19:08:27.502508427Z Waiting for performance profile collector pods to become ready: 3
2026-04-22T19:08:27.530399283Z Gathering data for ns/openshift-ingress-canary...
2026-04-22T19:08:27.878942641Z Gathering data for ns/openshift-insights...
2026-04-22T19:08:28.613082877Z Waiting for performance profile collector pods to become ready: 4
2026-04-22T19:08:29.747621571Z Waiting for performance profile collector pods to become ready: 5
2026-04-22T19:08:29.897077115Z Gathering data for ns/kserve...
2026-04-22T19:08:30.859869173Z Waiting for performance profile collector pods to become ready: 6
2026-04-22T19:08:31.309994064Z Gathering data for ns/openshift-config...
2026-04-22T19:08:31.542965215Z Gathering data for ns/openshift-config-managed...
2026-04-22T19:08:31.916874503Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-22T19:08:31.924714828Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-22T19:08:31.989599272Z Waiting for performance profile collector pods to become ready: 7
2026-04-22T19:08:32.168367192Z Gathering data for ns/openshift-kube-apiserver...
2026-04-22T19:08:32.447344160Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-22T19:08:32.741949745Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-22T19:08:33.002365418Z Gathering data for ns/openshift-kube-scheduler...
2026-04-22T19:08:33.119950702Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-22T19:08:33.266646464Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-22T19:08:33.398380029Z Collecting performance related data for node ip-10-0-133-42.ec2.internal
2026-04-22T19:08:33.405092756Z Collecting performance related data for node ip-10-0-128-208.ec2.internal
2026-04-22T19:08:33.412125703Z Collecting performance related data for node ip-10-0-136-85.ec2.internal
2026-04-22T19:08:33.626226409Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-22T19:08:34.059496423Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-22T19:08:34.435940685Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-22T19:08:35.026599325Z Gathering data for ns/openshift-multus...
2026-04-22T19:08:36.411832475Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-22T19:08:36.921413018Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-22T19:08:37.916393310Z Gathering data for ns/openshift-host-network...
2026-04-22T19:08:38.206357144Z Gathering data for ns/openshift-network-diagnostics...
2026-04-22T19:08:38.698995075Z Gathering data for ns/openshift-network-node-identity...
2026-04-22T19:08:39.088090979Z Gathering data for ns/openshift-network-operator...
2026-04-22T19:08:39.426703545Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-22T19:08:39.656380200Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-22T19:08:40.057693836Z Gathering data for ns/openshift-apiserver-operator...
2026-04-22T19:08:40.332028363Z Gathering data for ns/openshift-apiserver...
2026-04-22T19:08:40.833991966Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-22T19:08:41.107531387Z Gathering data for ns/openshift-controller-manager...
2026-04-22T19:08:41.343711453Z Collecting kubelet logs for node ip-10-0-133-42.ec2.internal
2026-04-22T19:08:41.346127445Z Collecting kubelet logs for node ip-10-0-128-208.ec2.internal
2026-04-22T19:08:41.347917670Z Collecting kubelet logs for node ip-10-0-136-85.ec2.internal
2026-04-22T19:08:41.353922708Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-22T19:08:41.726787349Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-22T19:08:41.731627033Z INFO: Node performance data collection complete.
2026-04-22T19:08:41.925867277Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-22T19:08:42.046497372Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-22T19:08:42.342481101Z Gathering data for ns/openshift-service-ca-operator...
2026-04-22T19:08:42.616647763Z Gathering data for ns/openshift-service-ca...
2026-04-22T19:08:42.934695638Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-22T19:08:43.744161372Z Wrote inspect data to must-gather.
2026-04-22T19:08:43.744207747Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T19:08:43.744207747Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-22T19:08:43.872095820Z error: the server doesn't have a resource type "clusters"
2026-04-22T19:08:44.014437413Z Caches written to disk