2026-04-21T02:04:37.812584506Z [disk usage checker] Started
2026-04-21T02:04:37.845616396Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T02:04:38.044169228Z Gathering data for ns/openshift-cluster-version...
2026-04-21T02:04:38.198407811Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T02:04:38.295867690Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T02:04:38.394273654Z Gathering data for ns/default...
2026-04-21T02:04:38.734116777Z Gathering data for ns/openshift...
2026-04-21T02:04:39.279362198Z Gathering data for ns/kube-system...
2026-04-21T02:04:39.962935091Z Gathering data for ns/openshift-etcd...
2026-04-21T02:04:40.315262820Z Wrote inspect data to must-gather.
2026-04-21T02:04:40.315301492Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T02:04:40.315301492Z     namespaces "assisted-installer" not found
2026-04-21T02:04:40.368710925Z Waiting on subprocesses to finish execution.
2026-04-21T02:04:40.373213030Z INFO: Gathering HAProxy config files
2026-04-21T02:04:40.375694987Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-21T02:04:40.390938138Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-21T02:04:40.391169339Z INFO: Collecting host service logs for crio
2026-04-21T02:04:40.391473121Z INFO: Collecting host service logs for kubelet
2026-04-21T02:04:40.391776301Z INFO: Collecting host service logs for rpm-ostreed
2026-04-21T02:04:40.392109224Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-21T02:04:40.392418993Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-21T02:04:40.392706266Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-21T02:04:40.393027248Z INFO: Collecting host service logs for NetworkManager
2026-04-21T02:04:40.393481893Z INFO: Collecting host service logs for openvswitch
2026-04-21T02:04:40.393589612Z INFO: Collecting host service logs for ovs-configuration
2026-04-21T02:04:40.393916516Z INFO: Collecting host service logs for ovsdb-server
2026-04-21T02:04:40.394207650Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-21T02:04:40.394571350Z INFO: Waiting for worker host service log collection to complete ...
2026-04-21T02:04:40.406310251Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-21T02:04:40.426476353Z INFO: Waiting for node performance related collection to complete ...
2026-04-21T02:04:40.874898866Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-21T02:04:40.875016024Z INFO: on-disk MachineConfig config collection complete.
2026-04-21T02:04:41.053989968Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-21T02:04:41.091686393Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-21T02:04:41.197108037Z No resources found
2026-04-21T02:04:41.501011137Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-cqf6k
2026-04-21T02:04:41.501065208Z insights-runtime-extractor-6ghj7
2026-04-21T02:04:41.501086056Z insights-runtime-extractor-j5d6m
2026-04-21T02:04:41.501105677Z insights-runtime-extractor-pbql6
2026-04-21T02:04:41.523478061Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-21T02:04:41.550963025Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-21T02:04:41.873831462Z No resources found in openshift-etcd namespace.
2026-04-21T02:04:41.944493754Z error: the server doesn't have a resource type "performanceprofile"
2026-04-21T02:04:42.008617757Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-21T02:04:42.115424702Z INFO: "metallb-operator" not detected. Skipping.
2026-04-21T02:04:42.312489262Z Wrote inspect data to must-gather.
2026-04-21T02:04:42.506974343Z INFO: Worker host service log collection to complete.
2026-04-21T02:04:42.548157764Z Wrote inspect data to must-gather.
2026-04-21T02:04:42.596486215Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-21T02:04:42.607694038Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T02:04:42.699428553Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-21T02:04:42.699428553Z See 'oc rsync -h' for help and examples
2026-04-21T02:04:42.704884121Z No resources found
2026-04-21T02:04:42.715915926Z INFO: 'previous-logs' folder not found on ip-10-0-129-42.ec2.internal, skipping...
2026-04-21T02:04:42.832211337Z ERROR: No running kube-apiserver pods found
2026-04-21T02:04:42.877677038Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-21T02:04:42.935483798Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T02:04:42.956746849Z Wrote inspect data to must-gather.
2026-04-21T02:04:43.130094664Z Gathering data for ns/kuadrant-system...
2026-04-21T02:04:43.134634800Z Wrote inspect data to must-gather.
2026-04-21T02:04:43.141039964Z Wrote inspect data to must-gather.
2026-04-21T02:04:43.191019881Z Wrote inspect data to must-gather.
2026-04-21T02:04:43.276574071Z tar: Removing leading `/' from member names
2026-04-21T02:04:43.280408764Z INFO: HAProxy config collection complete.
2026-04-21T02:04:43.319279785Z error: the server doesn't have a resource type "machineconfigs"
2026-04-21T02:04:43.320021893Z INFO: 'previous-logs' folder not found on ip-10-0-129-52.ec2.internal, skipping...
2026-04-21T02:04:43.488125293Z INFO: Getting rules from prometheus-k8s-0
2026-04-21T02:04:43.588433724Z Wrote inspect data to must-gather.
2026-04-21T02:04:43.668376028Z Wrote inspect data to must-gather.
2026-04-21T02:04:43.745097538Z INFO: 'previous-logs' folder not found on ip-10-0-141-35.ec2.internal, skipping...
2026-04-21T02:04:43.745208477Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-21T02:04:43.745261658Z INFO: Machine Config Daemon termination log collection complete.
2026-04-21T02:04:43.985105991Z Wrote inspect data to must-gather.
2026-04-21T02:04:43.990714088Z INFO: Getting status/config from prometheus-k8s-0
2026-04-21T02:04:43.995619704Z Wrote inspect data to must-gather.
2026-04-21T02:04:44.010327794Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-21T02:04:44.212576295Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-21T02:04:44.373993537Z Wrote inspect data to must-gather.
2026-04-21T02:04:44.489580934Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-21T02:04:44.509777908Z W0421 02:04:44.509737    1118 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-21T02:04:44.534688339Z Wrote inspect data to must-gather.
2026-04-21T02:04:44.619058059Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-21T02:04:44.752059260Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-21T02:04:44.757554313Z INFO: INTERCONNECT MODE
2026-04-21T02:04:44.757653514Z INFO: Gathering ovn-kubernetes DBs
2026-04-21T02:04:44.842989328Z Gathering data for ns/openshift-monitoring...
2026-04-21T02:04:44.948876672Z Wrote inspect data to must-gather.
2026-04-21T02:04:44.984761449Z INFO: Gathering OVN_Northbound from ovnkube-node-bvzx2...
2026-04-21T02:04:44.985009895Z INFO: Gathering OVN_Northbound from ovnkube-node-d79xd...
2026-04-21T02:04:44.985247226Z INFO: Gathering OVN_Northbound from ovnkube-node-f2tdx...
2026-04-21T02:04:44.986032016Z INFO: Gathering OVN_Southbound from ovnkube-node-bvzx2...
2026-04-21T02:04:44.986331172Z INFO: Gathering OVN_Southbound from ovnkube-node-d79xd...
2026-04-21T02:04:44.986611300Z INFO: Gathering OVN_Southbound from ovnkube-node-f2tdx...
2026-04-21T02:04:45.057000919Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-21T02:04:45.285118969Z tar: Removing leading `/' from member names
2026-04-21T02:04:45.293578744Z tar: Removing leading `/' from member names
2026-04-21T02:04:45.311573363Z tar: Removing leading `/' from member names
2026-04-21T02:04:45.369430314Z tar: Removing leading `/' from member names
2026-04-21T02:04:45.436375634Z tar: Removing leading `/' from member names
2026-04-21T02:04:45.443623722Z tar: Removing leading `/' from member names
2026-04-21T02:04:45.498946215Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-21T02:04:45.499157294Z Wrote inspect data to must-gather.
2026-04-21T02:04:45.911495892Z Wrote inspect data to must-gather.
2026-04-21T02:04:45.973308301Z INFO: Getting status from alertmanager-main-0
2026-04-21T02:04:46.073074715Z INFO: Waiting for network log collection to complete ...
2026-04-21T02:04:46.073121893Z INFO: Waiting for ovnk database copies to complete ...
2026-04-21T02:04:46.073591951Z INFO: Copying ovnk databases complete.
2026-04-21T02:04:46.077490069Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-04-21T02:04:46.089845438Z ovnk_database_store/
2026-04-21T02:04:46.089911810Z ovnk_database_store/ovnkube-node-f2tdx_nbdb
2026-04-21T02:04:46.111022106Z ovnk_database_store/ovnkube-node-bvzx2_sbdb
2026-04-21T02:04:46.149171235Z ovnk_database_store/ovnkube-node-bvzx2_nbdb
2026-04-21T02:04:46.170501076Z ovnk_database_store/ovnkube-node-d79xd_nbdb
2026-04-21T02:04:46.181596753Z ovnk_database_store/ovnkube-node-d79xd_sbdb
2026-04-21T02:04:46.208282246Z ovnk_database_store/ovnkube-node-f2tdx_sbdb
2026-04-21T02:04:46.243441152Z INFO: Network log collection complete.
2026-04-21T02:04:46.321403117Z Wrote inspect data to must-gather.
2026-04-21T02:04:46.612642926Z Wrote inspect data to must-gather.
2026-04-21T02:04:46.929994500Z W0421 02:04:46.929941    1427 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-21T02:04:46.943884291Z Wrote inspect data to must-gather.
2026-04-21T02:04:47.198096454Z Gathering data for ns/openshift-network-console...
2026-04-21T02:04:47.270491335Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-21T02:04:47.274963009Z INFO: Fallback to identify the container image from release info
2026-04-21T02:04:47.691669227Z Gathering data for ns/openshift-console-operator...
2026-04-21T02:04:47.864576906Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-21T02:04:47.944033607Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T02:04:48.013221630Z daemonset.apps/perf-node-gather-daemonset created
2026-04-21T02:04:48.195008587Z Gathering data for ns/openshift-console...
2026-04-21T02:04:48.202679854Z Waiting for performance profile collector pods to become ready: 1
2026-04-21T02:04:48.719227610Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-21T02:04:49.123633532Z Gathering data for ns/openshift-dns-operator...
2026-04-21T02:04:49.327682671Z Waiting for performance profile collector pods to become ready: 2
2026-04-21T02:04:49.513272717Z Gathering data for ns/openshift-dns...
2026-04-21T02:04:50.118581899Z Wrote inspect data to must-gather.
2026-04-21T02:04:50.227096659Z Gathering data for ns/openshift-image-registry...
2026-04-21T02:04:50.462970373Z Waiting for performance profile collector pods to become ready: 3
2026-04-21T02:04:50.689708742Z Gathering data for ns/openshift-ingress-operator...
2026-04-21T02:04:51.030535463Z Gathering data for ns/openshift-ingress...
2026-04-21T02:04:51.590232045Z Waiting for performance profile collector pods to become ready: 4
2026-04-21T02:04:51.656529161Z Gathering data for ns/openshift-ingress-canary...
2026-04-21T02:04:52.094347748Z Gathering data for ns/openshift-insights...
2026-04-21T02:04:52.725299328Z Waiting for performance profile collector pods to become ready: 5
2026-04-21T02:04:52.949076394Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T02:04:53.845318228Z Waiting for performance profile collector pods to become ready: 6
2026-04-21T02:04:54.120928444Z Gathering data for ns/opendatahub...
2026-04-21T02:04:54.969077947Z Waiting for performance profile collector pods to become ready: 7
2026-04-21T02:04:55.519328845Z Gathering data for ns/openshift-lws-operator...
2026-04-21T02:04:56.085337021Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-21T02:04:56.336399051Z Collecting performance related data for node ip-10-0-141-35.ec2.internal
2026-04-21T02:04:56.343462651Z Collecting performance related data for node ip-10-0-129-52.ec2.internal
2026-04-21T02:04:56.351134070Z Collecting performance related data for node ip-10-0-129-42.ec2.internal
2026-04-21T02:04:56.794883936Z Gathering data for ns/openshift-config...
2026-04-21T02:04:57.162246457Z Gathering data for ns/openshift-config-managed...
2026-04-21T02:04:57.741698076Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-21T02:04:57.957303020Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T02:04:58.091576470Z Gathering data for ns/openshift-kube-apiserver...
2026-04-21T02:04:58.403883991Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-21T02:04:58.769098646Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-21T02:04:59.142343186Z Gathering data for ns/openshift-kube-scheduler...
2026-04-21T02:04:59.475345509Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-21T02:04:59.802973474Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-21T02:05:00.141025641Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-21T02:05:00.472543768Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-21T02:05:01.081891120Z Gathering data for ns/openshift-multus...
2026-04-21T02:05:02.520242329Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-21T02:05:02.963330360Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T02:05:03.934395611Z Gathering data for ns/openshift-host-network...
2026-04-21T02:05:04.266511375Z Gathering data for ns/openshift-network-diagnostics...
2026-04-21T02:05:04.825604760Z Gathering data for ns/openshift-network-node-identity...
2026-04-21T02:05:04.941731034Z Collecting kubelet logs for node ip-10-0-141-35.ec2.internal
2026-04-21T02:05:04.943508390Z Collecting kubelet logs for node ip-10-0-129-52.ec2.internal
2026-04-21T02:05:04.945271366Z Collecting kubelet logs for node ip-10-0-129-42.ec2.internal
2026-04-21T02:05:05.360234992Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-21T02:05:05.365112657Z INFO: Node performance data collection complete.
2026-04-21T02:05:05.385727263Z Gathering data for ns/openshift-network-operator...
2026-04-21T02:05:05.787367465Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-21T02:05:06.082998318Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-21T02:05:06.603850551Z Gathering data for ns/openshift-apiserver-operator...
2026-04-21T02:05:06.921614152Z Gathering data for ns/openshift-apiserver...
2026-04-21T02:05:07.359203011Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-21T02:05:07.639073578Z Gathering data for ns/openshift-controller-manager...
2026-04-21T02:05:07.945831653Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-21T02:05:07.968977856Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-04-21T02:05:08.590142340Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-21T02:05:08.885552900Z Gathering data for ns/openshift-service-ca-operator...
2026-04-21T02:05:09.205633322Z Gathering data for ns/openshift-service-ca...
2026-04-21T02:05:09.589222452Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-21T02:05:10.555249404Z Wrote inspect data to must-gather.
2026-04-21T02:05:10.555299558Z error: inspection completed with the errors occurred while gathering data:
2026-04-21T02:05:10.555299558Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-21T02:05:10.692458263Z error: the server doesn't have a resource type "clusters"
2026-04-21T02:05:10.869201016Z Caches written to disk