2026-04-20T15:38:37.737001620Z [disk usage checker] Started 2026-04-20T15:38:37.741094309Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-04-20T15:38:37.974879913Z Gathering data for ns/openshift-cluster-version... 2026-04-20T15:38:38.131643685Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-20T15:38:38.223976434Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-20T15:38:38.331023061Z Gathering data for ns/default... 2026-04-20T15:38:38.688903010Z Gathering data for ns/openshift... 2026-04-20T15:38:39.163559392Z Gathering data for ns/kube-system... 2026-04-20T15:38:39.853952556Z Gathering data for ns/openshift-etcd... 2026-04-20T15:38:40.211059278Z Wrote inspect data to must-gather. 2026-04-20T15:38:40.211183309Z error: inspection completed with the errors occurred while gathering data: 2026-04-20T15:38:40.211183309Z namespaces "assisted-installer" not found 2026-04-20T15:38:40.264559222Z Waiting on subprocesses to finish execution. 2026-04-20T15:38:40.273550539Z INFO: Gathering machine config daemon's old logs from all nodes 2026-04-20T15:38:40.274565563Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-04-20T15:38:40.276481394Z INFO: Gathering HAProxy config files 2026-04-20T15:38:40.284100285Z INFO: Collecting host service logs for crio 2026-04-20T15:38:40.284224111Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-04-20T15:38:40.284484022Z INFO: Collecting host service logs for kubelet 2026-04-20T15:38:40.284803571Z INFO: Collecting host service logs for rpm-ostreed 2026-04-20T15:38:40.285112770Z INFO: Collecting host service logs for ostree-finalize-staged 2026-04-20T15:38:40.285449124Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-04-20T15:38:40.285753158Z INFO: Collecting host service logs for machine-config-daemon-host 2026-04-20T15:38:40.286051353Z INFO: Collecting host service logs for NetworkManager 2026-04-20T15:38:40.286381180Z INFO: Collecting host service logs for openvswitch 2026-04-20T15:38:40.286682117Z INFO: Collecting host service logs for ovs-configuration 2026-04-20T15:38:40.286982857Z INFO: Collecting host service logs for ovsdb-server 2026-04-20T15:38:40.287315388Z INFO: Collecting host service logs for ovs-vswitchd 2026-04-20T15:38:40.287682180Z INFO: Waiting for worker host service log collection to complete ... 2026-04-20T15:38:40.297485677Z INFO: Waiting for node performance related collection to complete ... 2026-04-20T15:38:40.775548610Z No resources found 2026-04-20T15:38:40.925382420Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-04-20T15:38:40.925418109Z INFO: "sriov-network-operator" not detected. Skipping. 2026-04-20T15:38:40.925471834Z INFO: on-disk MachineConfig config collection complete. 2026-04-20T15:38:40.993578201Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-04-20T15:38:41.120291633Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-mgv7v 2026-04-20T15:38:41.120367274Z insights-runtime-extractor-6jz9t 2026-04-20T15:38:41.120389277Z insights-runtime-extractor-7qmvr 2026-04-20T15:38:41.120409178Z insights-runtime-extractor-8mrp8 2026-04-20T15:38:41.350041468Z No resources found in openshift-etcd namespace. 2026-04-20T15:38:41.386035721Z Wrote inspect data to must-gather. 2026-04-20T15:38:41.428006064Z error: the server doesn't have a resource type "performanceprofile" 2026-04-20T15:38:41.528157576Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-04-20T15:38:41.598350370Z INFO: Worker host service log collection to complete. 2026-04-20T15:38:41.616842449Z INFO: "metallb-operator" not detected. Skipping. 2026-04-20T15:38:41.629092653Z INFO: Found 1 replicas - prometheus-k8s-0 2026-04-20T15:38:41.827838945Z INFO: Waiting for HAProxy config collection to complete ... 2026-04-20T15:38:41.873917905Z ERROR: No running kube-apiserver pods found 2026-04-20T15:38:41.886446646Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-04-20T15:38:41.886446646Z See 'oc rsync -h' for help and examples 2026-04-20T15:38:42.185374156Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-04-20T15:38:42.186525999Z Wrote inspect data to must-gather. 2026-04-20T15:38:42.427678001Z tar: Removing leading `/' from member names 2026-04-20T15:38:42.436337989Z INFO: HAProxy config collection complete. 2026-04-20T15:38:42.502682427Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-04-20T15:38:42.577512599Z INFO: 'previous-logs' folder not found on ip-10-0-129-115.ec2.internal, skipping... 2026-04-20T15:38:42.931945253Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-04-20T15:38:43.009141613Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-20T15:38:43.060246403Z Wrote inspect data to must-gather. 2026-04-20T15:38:43.075453980Z Wrote inspect data to must-gather. 2026-04-20T15:38:43.128586821Z error: the server doesn't have a resource type "machineconfigs" 2026-04-20T15:38:43.137482999Z Wrote inspect data to must-gather. 2026-04-20T15:38:43.159073451Z No resources found 2026-04-20T15:38:43.203629341Z Wrote inspect data to must-gather. 2026-04-20T15:38:43.265512091Z INFO: 'previous-logs' folder not found on ip-10-0-133-198.ec2.internal, skipping... 2026-04-20T15:38:43.316314820Z INFO: Getting rules from prometheus-k8s-0 2026-04-20T15:38:43.416393973Z Gathering data for ns/kuadrant-system... 2026-04-20T15:38:43.485512113Z Wrote inspect data to must-gather. 2026-04-20T15:38:43.570065988Z Wrote inspect data to must-gather. 2026-04-20T15:38:43.642981357Z INFO: 'previous-logs' folder not found on ip-10-0-134-230.ec2.internal, skipping... 2026-04-20T15:38:43.643109079Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-04-20T15:38:43.643168917Z INFO: Machine Config Daemon termination log collection complete. 2026-04-20T15:38:43.780397623Z Wrote inspect data to must-gather. 2026-04-20T15:38:43.877409761Z INFO: Getting status/config from prometheus-k8s-0 2026-04-20T15:38:43.916746154Z Wrote inspect data to must-gather. 2026-04-20T15:38:44.091539221Z error: the server doesn't have a resource type "kubeletconfigs" 2026-04-20T15:38:44.355079852Z Wrote inspect data to must-gather. 2026-04-20T15:38:44.401516128Z Wrote inspect data to must-gather. 2026-04-20T15:38:44.406490100Z INFO: Getting status/flags from prometheus-k8s-0 2026-04-20T15:38:44.727466612Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-04-20T15:38:44.757632709Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ 2026-04-20T15:38:44.760058084Z Wrote inspect data to must-gather. 2026-04-20T15:38:44.770416547Z INFO: INTERCONNECT MODE 2026-04-20T15:38:44.770457198Z INFO: Gathering ovn-kubernetes DBs 2026-04-20T15:38:45.014643293Z INFO: Gathering OVN_Northbound from ovnkube-node-nhq74... 2026-04-20T15:38:45.014880022Z INFO: Gathering OVN_Northbound from ovnkube-node-sdgqx... 2026-04-20T15:38:45.015100074Z INFO: Gathering OVN_Northbound from ovnkube-node-x9pxn... 2026-04-20T15:38:45.015444463Z INFO: Gathering OVN_Southbound from ovnkube-node-nhq74... 2026-04-20T15:38:45.015729567Z INFO: Gathering OVN_Southbound from ovnkube-node-sdgqx... 2026-04-20T15:38:45.015990983Z INFO: Gathering OVN_Southbound from ovnkube-node-x9pxn... 2026-04-20T15:38:45.038795678Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-04-20T15:38:45.101387622Z Wrote inspect data to must-gather. 2026-04-20T15:38:45.338094320Z tar: Removing leading `/' from member names 2026-04-20T15:38:45.403215719Z tar: Removing leading `/' from member names 2026-04-20T15:38:45.414442406Z tar: Removing leading `/' from member names 2026-04-20T15:38:45.478330439Z tar: Removing leading `/' from member names 2026-04-20T15:38:45.513523635Z tar: Removing leading `/' from member names 2026-04-20T15:38:45.528625157Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-04-20T15:38:45.535458731Z tar: Removing leading `/' from member names 2026-04-20T15:38:45.550569620Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-04-20T15:38:45.727134253Z Wrote inspect data to must-gather. 2026-04-20T15:38:45.827066059Z Gathering data for ns/openshift-monitoring... 2026-04-20T15:38:46.060046043Z INFO: Getting status from alertmanager-main-0 2026-04-20T15:38:46.174248038Z INFO: Waiting for network log collection to complete ... 2026-04-20T15:38:46.174311541Z INFO: Waiting for ovnk database copies to complete ... 2026-04-20T15:38:46.174823029Z INFO: Copying ovnk databases complete. 2026-04-20T15:38:46.176621707Z 4.1M must-gather/network_logs/ovnk_database_store 2026-04-20T15:38:46.179120874Z ovnk_database_store/ 2026-04-20T15:38:46.179174460Z ovnk_database_store/ovnkube-node-x9pxn_nbdb 2026-04-20T15:38:46.190111994Z Wrote inspect data to must-gather. 2026-04-20T15:38:46.200369579Z ovnk_database_store/ovnkube-node-nhq74_nbdb 2026-04-20T15:38:46.212950444Z ovnk_database_store/ovnkube-node-sdgqx_sbdb 2026-04-20T15:38:46.236258883Z ovnk_database_store/ovnkube-node-x9pxn_sbdb 2026-04-20T15:38:46.265968054Z ovnk_database_store/ovnkube-node-sdgqx_nbdb 2026-04-20T15:38:46.279089635Z ovnk_database_store/ovnkube-node-nhq74_sbdb 2026-04-20T15:38:46.311579279Z INFO: Network log collection complete. 2026-04-20T15:38:46.510499417Z Wrote inspect data to must-gather. 2026-04-20T15:38:46.814021602Z Wrote inspect data to must-gather. 2026-04-20T15:38:47.123179537Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found 2026-04-20T15:38:47.125448772Z INFO: Fallback to identify the container image from release info 2026-04-20T15:38:47.713611389Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d 2026-04-20T15:38:47.848524323Z daemonset.apps/perf-node-gather-daemonset created 2026-04-20T15:38:47.925672214Z Gathering data for ns/openshift-network-console... 2026-04-20T15:38:47.947941055Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-04-20T15:38:48.008800610Z Waiting for performance profile collector pods to become ready: 1 2026-04-20T15:38:48.455659014Z Gathering data for ns/openshift-console-operator... 2026-04-20T15:38:48.938309520Z Gathering data for ns/openshift-console... 2026-04-20T15:38:49.166307506Z Waiting for performance profile collector pods to become ready: 2 2026-04-20T15:38:49.483500395Z Gathering data for ns/openshift-cluster-storage-operator... 2026-04-20T15:38:49.885825097Z Wrote inspect data to must-gather. 2026-04-20T15:38:49.887571757Z Gathering data for ns/openshift-dns-operator... 2026-04-20T15:38:50.215421268Z Gathering data for ns/openshift-dns... 2026-04-20T15:38:50.288245213Z Waiting for performance profile collector pods to become ready: 3 2026-04-20T15:38:50.814714440Z Gathering data for ns/openshift-image-registry... 2026-04-20T15:38:51.284444130Z Gathering data for ns/openshift-ingress-operator... 2026-04-20T15:38:51.419966784Z Waiting for performance profile collector pods to become ready: 4 2026-04-20T15:38:51.700281953Z Gathering data for ns/openshift-ingress... 2026-04-20T15:38:52.409042483Z Gathering data for ns/openshift-ingress-canary... 2026-04-20T15:38:52.545821555Z Waiting for performance profile collector pods to become ready: 5 2026-04-20T15:38:52.893043159Z Gathering data for ns/openshift-insights... 2026-04-20T15:38:52.952475011Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-04-20T15:38:53.664174888Z Waiting for performance profile collector pods to become ready: 6 2026-04-20T15:38:54.782933476Z Waiting for performance profile collector pods to become ready: 7 2026-04-20T15:38:54.825549810Z Gathering data for ns/opendatahub... 2026-04-20T15:38:55.900891998Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-04-20T15:38:56.146090554Z Collecting performance related data for node ip-10-0-133-198.ec2.internal 2026-04-20T15:38:56.153170617Z Collecting performance related data for node ip-10-0-129-115.ec2.internal 2026-04-20T15:38:56.161612955Z Collecting performance related data for node ip-10-0-134-230.ec2.internal 2026-04-20T15:38:56.308521405Z Gathering data for ns/openshift-lws-operator... 2026-04-20T15:38:57.785694844Z Gathering data for ns/openshift-config... 2026-04-20T15:38:57.961206784Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-04-20T15:38:58.079308083Z Gathering data for ns/openshift-config-managed... 2026-04-20T15:38:58.588907501Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-04-20T15:38:58.937760508Z Gathering data for ns/openshift-kube-apiserver... 2026-04-20T15:38:59.385048625Z Gathering data for ns/openshift-kube-controller-manager... 2026-04-20T15:38:59.729718804Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-04-20T15:39:00.051228643Z Gathering data for ns/openshift-kube-scheduler... 2026-04-20T15:39:00.340820045Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-04-20T15:39:00.696048939Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-04-20T15:39:01.057875255Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-04-20T15:39:01.400701210Z Gathering data for ns/openshift-user-workload-monitoring... 2026-04-20T15:39:01.958359029Z Gathering data for ns/openshift-multus... 2026-04-20T15:39:02.965963084Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-04-20T15:39:03.621906094Z Gathering data for ns/openshift-ovn-kubernetes... 2026-04-20T15:39:04.788681722Z Collecting kubelet logs for node ip-10-0-133-198.ec2.internal 2026-04-20T15:39:04.790630294Z Collecting kubelet logs for node ip-10-0-129-115.ec2.internal 2026-04-20T15:39:04.792341198Z Collecting kubelet logs for node ip-10-0-134-230.ec2.internal 2026-04-20T15:39:05.160147489Z daemonset.apps "perf-node-gather-daemonset" deleted 2026-04-20T15:39:05.166798487Z INFO: Node performance data collection complete. 2026-04-20T15:39:05.484204759Z Gathering data for ns/openshift-host-network... 2026-04-20T15:39:05.870555521Z Gathering data for ns/openshift-network-diagnostics... 2026-04-20T15:39:06.405448163Z Gathering data for ns/openshift-network-node-identity... 2026-04-20T15:39:06.837667856Z Gathering data for ns/openshift-network-operator... 2026-04-20T15:39:07.235966888Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-04-20T15:39:07.527387003Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-04-20T15:39:07.972524700Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70 2026-04-20T15:39:08.009422017Z Gathering data for ns/openshift-apiserver-operator... 2026-04-20T15:39:08.374137144Z Gathering data for ns/openshift-apiserver... 2026-04-20T15:39:08.826550420Z Gathering data for ns/openshift-controller-manager-operator... 2026-04-20T15:39:09.122787108Z Gathering data for ns/openshift-controller-manager... 2026-04-20T15:39:09.438550240Z Gathering data for ns/openshift-cluster-samples-operator... 2026-04-20T15:39:10.086225114Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-04-20T15:39:10.387953253Z Gathering data for ns/openshift-service-ca-operator... 2026-04-20T15:39:10.720998254Z Gathering data for ns/openshift-service-ca... 2026-04-20T15:39:11.122006529Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-04-20T15:39:12.118243623Z Wrote inspect data to must-gather. 2026-04-20T15:39:12.118477681Z error: inspection completed with the errors occurred while gathering data: 2026-04-20T15:39:12.118477681Z skipping gathering secrets/support due to error: secrets "support" not found 2026-04-20T15:39:12.254023534Z error: the server doesn't have a resource type "clusters" 2026-04-20T15:39:12.432558868Z Caches written to disk