2026-04-16T19:42:51.970097098Z [disk usage checker] Started
2026-04-16T19:42:52.000834017Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:42:52.207568206Z Gathering data for ns/openshift-cluster-version...
2026-04-16T19:42:52.370877270Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:42:52.463277568Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:42:52.555899676Z Gathering data for ns/default...
2026-04-16T19:42:52.910693965Z Gathering data for ns/openshift...
2026-04-16T19:42:53.376969909Z Gathering data for ns/kube-system...
2026-04-16T19:42:54.102843479Z Gathering data for ns/openshift-etcd...
2026-04-16T19:42:54.379612369Z Waiting on subprocesses to finish execution.
2026-04-16T19:42:54.388526704Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T19:42:54.391152355Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T19:42:54.393551653Z INFO: Collecting host service logs for crio
2026-04-16T19:42:54.393888877Z INFO: Collecting host service logs for kubelet
2026-04-16T19:42:54.394305297Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T19:42:54.394635757Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T19:42:54.394966715Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T19:42:54.395303996Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T19:42:54.395633257Z INFO: Collecting host service logs for NetworkManager
2026-04-16T19:42:54.395957943Z INFO: Collecting host service logs for openvswitch
2026-04-16T19:42:54.396313576Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T19:42:54.396663251Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T19:42:54.396985020Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T19:42:54.397412741Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T19:42:54.399554429Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T19:42:54.403975642Z INFO: Gathering HAProxy config files
2026-04-16T19:42:54.594621208Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T19:42:54.705571093Z Wrote inspect data to must-gather.
2026-04-16T19:42:54.705612989Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:42:54.705612989Z     namespaces "assisted-installer" not found
2026-04-16T19:42:54.854169087Z No resources found in openshift-etcd namespace.
2026-04-16T19:42:54.989467972Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T19:42:55.240970937Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T19:42:55.519253093Z No resources found
2026-04-16T19:42:55.687221088Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-5r8bs
2026-04-16T19:42:55.687273618Z insights-runtime-extractor-f9rdz
2026-04-16T19:42:55.687293264Z insights-runtime-extractor-nb8l9
2026-04-16T19:42:55.687312989Z insights-runtime-extractor-xzqwj
2026-04-16T19:42:55.759283652Z ERROR: No running kube-apiserver pods found
2026-04-16T19:42:55.819854151Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T19:42:55.960246473Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T19:42:56.150314031Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T19:42:56.150416619Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T19:42:56.180602298Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T19:42:56.259285260Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T19:42:56.372626963Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T19:42:56.408045272Z INFO: Worker host service log collection to complete.
2026-04-16T19:42:56.462315477Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T19:42:56.463409753Z Wrote inspect data to must-gather.
2026-04-16T19:42:56.551781675Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T19:42:56.551781675Z See 'oc rsync -h' for help and examples
2026-04-16T19:42:56.573815921Z Wrote inspect data to must-gather.
2026-04-16T19:42:56.625545577Z No resources found
2026-04-16T19:42:56.752789193Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T19:42:56.781329200Z INFO: 'previous-logs' folder not found on ip-10-0-129-155.ec2.internal, skipping...
2026-04-16T19:42:56.821439360Z tar: Removing leading `/' from member names
2026-04-16T19:42:56.827427255Z INFO: HAProxy config collection complete.
2026-04-16T19:42:56.829587282Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:42:56.855657307Z Wrote inspect data to must-gather.
2026-04-16T19:42:56.972439056Z Wrote inspect data to must-gather.
2026-04-16T19:42:57.023446624Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:42:57.031597299Z Wrote inspect data to must-gather.
2026-04-16T19:42:57.117966161Z Wrote inspect data to must-gather.
2026-04-16T19:42:57.188647997Z Gathering data for ns/kuadrant-system...
2026-04-16T19:42:57.301023147Z INFO: 'previous-logs' folder not found on ip-10-0-133-198.ec2.internal, skipping...
2026-04-16T19:42:57.328618062Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T19:42:57.373044214Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T19:42:57.393477805Z Wrote inspect data to must-gather.
2026-04-16T19:42:57.667875768Z INFO: 'previous-logs' folder not found on ip-10-0-133-241.ec2.internal, skipping...
2026-04-16T19:42:57.667977586Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T19:42:57.668027025Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T19:42:57.708600600Z Wrote inspect data to must-gather.
2026-04-16T19:42:57.768837848Z Wrote inspect data to must-gather.
2026-04-16T19:42:57.824871322Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T19:42:58.005378940Z Wrote inspect data to must-gather.
2026-04-16T19:42:58.118139416Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T19:42:58.123426589Z Wrote inspect data to must-gather.
2026-04-16T19:42:58.195977319Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T19:42:58.316348099Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T19:42:58.504273035Z INFO: INTERCONNECT MODE
2026-04-16T19:42:58.504371289Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T19:42:58.573258065Z Wrote inspect data to must-gather.
2026-04-16T19:42:58.592855593Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T19:42:58.699663592Z INFO: Gathering OVN_Northbound from ovnkube-node-gxnqw...
2026-04-16T19:42:58.699946333Z INFO: Gathering OVN_Northbound from ovnkube-node-mx7dg...
2026-04-16T19:42:58.700291269Z INFO: Gathering OVN_Northbound from ovnkube-node-vf8sc...
2026-04-16T19:42:58.700582802Z INFO: Gathering OVN_Southbound from ovnkube-node-gxnqw...
2026-04-16T19:42:58.700874589Z INFO: Gathering OVN_Southbound from ovnkube-node-mx7dg...
2026-04-16T19:42:58.701185802Z INFO: Gathering OVN_Southbound from ovnkube-node-vf8sc...
2026-04-16T19:42:58.703384594Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T19:42:59.022983102Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T19:42:59.083485890Z tar: Removing leading `/' from member names
2026-04-16T19:42:59.092202330Z tar: Removing leading `/' from member names
2026-04-16T19:42:59.104248069Z Wrote inspect data to must-gather.
2026-04-16T19:42:59.129482961Z Gathering data for ns/openshift-monitoring...
2026-04-16T19:42:59.187105054Z tar: Removing leading `/' from member names
2026-04-16T19:42:59.204769253Z tar: Removing leading `/' from member names
2026-04-16T19:42:59.224772732Z tar: Removing leading `/' from member names
2026-04-16T19:42:59.254490816Z tar: Removing leading `/' from member names
2026-04-16T19:42:59.391692524Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T19:42:59.495413041Z Wrote inspect data to must-gather.
2026-04-16T19:42:59.723163167Z INFO: Waiting for network log collection to complete ...
2026-04-16T19:42:59.723198535Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T19:42:59.723595330Z INFO: Copying ovnk databases complete.
2026-04-16T19:42:59.725220165Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-04-16T19:42:59.727473545Z ovnk_database_store/
2026-04-16T19:42:59.727473545Z ovnk_database_store/ovnkube-node-gxnqw_nbdb
2026-04-16T19:42:59.740669331Z ovnk_database_store/ovnkube-node-vf8sc_sbdb
2026-04-16T19:42:59.768457232Z ovnk_database_store/ovnkube-node-mx7dg_sbdb
2026-04-16T19:42:59.799449157Z ovnk_database_store/ovnkube-node-mx7dg_nbdb
2026-04-16T19:42:59.811813173Z ovnk_database_store/ovnkube-node-gxnqw_sbdb
2026-04-16T19:42:59.836803494Z ovnk_database_store/ovnkube-node-vf8sc_nbdb
2026-04-16T19:42:59.839697627Z INFO: Getting status from alertmanager-main-0
2026-04-16T19:42:59.851351311Z INFO: Network log collection complete.
2026-04-16T19:42:59.856192759Z Wrote inspect data to must-gather.
2026-04-16T19:43:00.157788791Z Wrote inspect data to must-gather.
2026-04-16T19:43:00.472126978Z Wrote inspect data to must-gather.
2026-04-16T19:43:00.779618385Z Wrote inspect data to must-gather.
2026-04-16T19:43:01.037044356Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T19:43:01.039566117Z INFO: Fallback to identify the container image from release info
2026-04-16T19:43:01.414595947Z Gathering data for ns/openshift-network-console...
2026-04-16T19:43:01.745875234Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-16T19:43:01.890187919Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T19:43:01.926840380Z Gathering data for ns/openshift-console-operator...
2026-04-16T19:43:02.036069648Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:43:02.039255580Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T19:43:02.477869564Z Gathering data for ns/openshift-console...
2026-04-16T19:43:02.969319102Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T19:43:03.184098091Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T19:43:03.412249178Z Gathering data for ns/openshift-dns-operator...
2026-04-16T19:43:03.744813945Z Gathering data for ns/openshift-dns...
2026-04-16T19:43:03.820569725Z Wrote inspect data to must-gather.
2026-04-16T19:43:04.325971584Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T19:43:04.371026884Z Gathering data for ns/openshift-image-registry...
2026-04-16T19:43:04.840834104Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T19:43:05.212432358Z Gathering data for ns/openshift-ingress...
2026-04-16T19:43:05.436081426Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T19:43:05.853784439Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T19:43:06.327458976Z Gathering data for ns/openshift-insights...
2026-04-16T19:43:06.543170108Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T19:43:07.040393577Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:43:07.650251674Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T19:43:08.292783535Z Gathering data for ns/opendatahub...
2026-04-16T19:43:08.766175270Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T19:43:09.562300821Z Gathering data for ns/openshift-lws-operator...
2026-04-16T19:43:09.892658064Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T19:43:10.143711263Z Collecting performance related data for node ip-10-0-133-198.ec2.internal
2026-04-16T19:43:10.150644346Z Collecting performance related data for node ip-10-0-129-155.ec2.internal
2026-04-16T19:43:10.157772585Z Collecting performance related data for node ip-10-0-133-241.ec2.internal
2026-04-16T19:43:10.982185039Z Gathering data for ns/openshift-config...
2026-04-16T19:43:11.368313871Z Gathering data for ns/openshift-config-managed...
2026-04-16T19:43:11.860479369Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T19:43:12.044918561Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:43:12.175499266Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T19:43:12.499608826Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T19:43:12.877100387Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T19:43:13.239322966Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T19:43:13.546340801Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T19:43:13.892260787Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T19:43:14.242258299Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T19:43:14.584435030Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T19:43:15.164353651Z Gathering data for ns/openshift-multus...
2026-04-16T19:43:16.584825554Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T19:43:17.049310026Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:43:18.180850383Z Gathering data for ns/openshift-host-network...
2026-04-16T19:43:18.328464454Z Collecting kubelet logs for node ip-10-0-133-198.ec2.internal
2026-04-16T19:43:18.330664405Z Collecting kubelet logs for node ip-10-0-129-155.ec2.internal
2026-04-16T19:43:18.332497844Z Collecting kubelet logs for node ip-10-0-133-241.ec2.internal
2026-04-16T19:43:18.593761545Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T19:43:18.679741029Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T19:43:18.684501194Z INFO: Node performance data collection complete.
2026-04-16T19:43:19.175714996Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T19:43:19.625429933Z Gathering data for ns/openshift-network-operator...
2026-04-16T19:43:20.006696050Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T19:43:20.337771639Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T19:43:20.798424255Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T19:43:21.133388877Z Gathering data for ns/openshift-apiserver...
2026-04-16T19:43:21.556666143Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T19:43:21.830240899Z Gathering data for ns/openshift-controller-manager...
2026-04-16T19:43:22.055284415Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-16T19:43:22.164574791Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T19:43:22.803320876Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T19:43:23.125622100Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T19:43:23.447797846Z Gathering data for ns/openshift-service-ca...
2026-04-16T19:43:23.831737631Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T19:43:24.854510136Z Wrote inspect data to must-gather.
2026-04-16T19:43:24.854551380Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T19:43:24.854551380Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T19:43:24.985300982Z error: the server doesn't have a resource type "clusters"
2026-04-16T19:43:25.169079140Z Caches written to disk