2026-05-28T21:30:21.024487101Z [disk usage checker] Started
2026-05-28T21:30:21.028223987Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T21:30:21.250229551Z Gathering data for ns/openshift-cluster-version...
2026-05-28T21:30:21.407186028Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T21:30:21.499295483Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T21:30:21.597299445Z Gathering data for ns/default...
2026-05-28T21:30:21.946315755Z Gathering data for ns/openshift...
2026-05-28T21:30:22.423080445Z Gathering data for ns/kube-system...
2026-05-28T21:30:23.119874557Z Gathering data for ns/openshift-etcd...
2026-05-28T21:30:23.487445843Z Wrote inspect data to must-gather.
2026-05-28T21:30:23.487479153Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T21:30:23.487479153Z     namespaces "assisted-installer" not found
2026-05-28T21:30:23.518486159Z Waiting on subprocesses to finish execution.
2026-05-28T21:30:23.532307772Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T21:30:23.533458611Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T21:30:23.535052065Z INFO: Gathering HAProxy config files
2026-05-28T21:30:23.541180296Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T21:30:23.557851207Z INFO: Collecting host service logs for crio
2026-05-28T21:30:23.558195981Z INFO: Collecting host service logs for kubelet
2026-05-28T21:30:23.558528646Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T21:30:23.558873780Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T21:30:23.559204808Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T21:30:23.559535744Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T21:30:23.559877390Z INFO: Collecting host service logs for NetworkManager
2026-05-28T21:30:23.560244662Z INFO: Collecting host service logs for openvswitch
2026-05-28T21:30:23.560589872Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T21:30:23.560961614Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T21:30:23.561324417Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T21:30:23.561769862Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T21:30:23.737244540Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T21:30:23.952596815Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T21:30:24.415162352Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T21:30:24.415243704Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T21:30:24.479953035Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T21:30:24.617216664Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T21:30:24.635809305Z No resources found
2026-05-28T21:30:24.715885245Z No resources found in openshift-etcd namespace.
2026-05-28T21:30:24.800284253Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-fbw9p
2026-05-28T21:30:24.800446125Z insights-runtime-extractor-dpgcb
2026-05-28T21:30:24.800474795Z insights-runtime-extractor-gw2nm
2026-05-28T21:30:24.800498197Z insights-runtime-extractor-sns5f
2026-05-28T21:30:24.861637761Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T21:30:24.939810012Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T21:30:24.976303897Z INFO: Worker host service log collection to complete.
2026-05-28T21:30:24.990435062Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T21:30:25.005823748Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T21:30:25.035647326Z Wrote inspect data to must-gather.
2026-05-28T21:30:25.144484222Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T21:30:25.144484222Z See 'oc rsync -h' for help and examples
2026-05-28T21:30:25.216354535Z ERROR: No running kube-apiserver pods found
2026-05-28T21:30:25.500373622Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T21:30:25.589594182Z Wrote inspect data to must-gather.
2026-05-28T21:30:25.872997080Z INFO: 'previous-logs' folder not found on ip-10-0-133-41.ec2.internal, skipping...
2026-05-28T21:30:26.045434378Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T21:30:26.113713504Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T21:30:26.151550315Z No resources found
2026-05-28T21:30:26.174584674Z Wrote inspect data to must-gather.
2026-05-28T21:30:26.177933792Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T21:30:26.242567789Z Gathering data for ns/kuadrant-system...
2026-05-28T21:30:26.349831476Z Wrote inspect data to must-gather.
2026-05-28T21:30:26.386244885Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T21:30:26.395494072Z Wrote inspect data to must-gather.
2026-05-28T21:30:26.417722517Z Wrote inspect data to must-gather.
2026-05-28T21:30:26.439853315Z tar: Removing leading `/' from member names
2026-05-28T21:30:26.446382153Z INFO: HAProxy config collection complete.
2026-05-28T21:30:26.560794875Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T21:30:26.604112783Z INFO: 'previous-logs' folder not found on ip-10-0-136-77.ec2.internal, skipping...
2026-05-28T21:30:26.731196256Z Wrote inspect data to must-gather.
2026-05-28T21:30:26.823930423Z Wrote inspect data to must-gather.
2026-05-28T21:30:27.053580174Z INFO: 'previous-logs' folder not found on ip-10-0-140-94.ec2.internal, skipping...
2026-05-28T21:30:27.053617563Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T21:30:27.053640399Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T21:30:27.073235353Z Wrote inspect data to must-gather.
2026-05-28T21:30:27.099796720Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T21:30:27.262475176Z Wrote inspect data to must-gather.
2026-05-28T21:30:27.331447729Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T21:30:27.367637352Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T21:30:27.646025855Z Wrote inspect data to must-gather.
2026-05-28T21:30:27.649954427Z Wrote inspect data to must-gather.
2026-05-28T21:30:27.652611671Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T21:30:27.920176428Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T21:30:27.920358324Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T21:30:28.047365245Z Wrote inspect data to must-gather.
2026-05-28T21:30:28.068343448Z INFO: INTERCONNECT MODE
2026-05-28T21:30:28.068433995Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T21:30:28.155329088Z Gathering data for ns/openshift-monitoring...
2026-05-28T21:30:28.228297879Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T21:30:28.306728208Z INFO: Gathering OVN_Northbound from ovnkube-node-j8294...
2026-05-28T21:30:28.306969705Z INFO: Gathering OVN_Northbound from ovnkube-node-kq6z5...
2026-05-28T21:30:28.307199668Z INFO: Gathering OVN_Northbound from ovnkube-node-nlv9g...
2026-05-28T21:30:28.307518180Z INFO: Gathering OVN_Southbound from ovnkube-node-j8294...
2026-05-28T21:30:28.307814351Z INFO: Gathering OVN_Southbound from ovnkube-node-kq6z5...
2026-05-28T21:30:28.308274692Z INFO: Gathering OVN_Southbound from ovnkube-node-nlv9g...
2026-05-28T21:30:28.461179109Z Wrote inspect data to must-gather.
2026-05-28T21:30:28.596975004Z tar: Removing leading `/' from member names
2026-05-28T21:30:28.684993215Z tar: Removing leading `/' from member names
2026-05-28T21:30:28.721259366Z tar: Removing leading `/' from member names
2026-05-28T21:30:28.771394051Z tar: Removing leading `/' from member names
2026-05-28T21:30:28.798257239Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T21:30:28.857300414Z tar: Removing leading `/' from member names
2026-05-28T21:30:28.857300414Z tar: Removing leading `/' from member names
2026-05-28T21:30:28.969774413Z Wrote inspect data to must-gather.
2026-05-28T21:30:29.285770517Z INFO: Getting status from alertmanager-main-0
2026-05-28T21:30:29.349341644Z Wrote inspect data to must-gather.
2026-05-28T21:30:29.434360600Z INFO: Waiting for network log collection to complete ...
2026-05-28T21:30:29.434470520Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T21:30:29.435022431Z INFO: Copying ovnk databases complete.
2026-05-28T21:30:29.438533659Z 4.3M	must-gather/network_logs/ovnk_database_store
2026-05-28T21:30:29.441200750Z ovnk_database_store/
2026-05-28T21:30:29.441249340Z ovnk_database_store/ovnkube-node-nlv9g_nbdb
2026-05-28T21:30:29.452084727Z ovnk_database_store/ovnkube-node-nlv9g_sbdb
2026-05-28T21:30:29.484168806Z ovnk_database_store/ovnkube-node-j8294_nbdb
2026-05-28T21:30:29.499610887Z ovnk_database_store/ovnkube-node-j8294_sbdb
2026-05-28T21:30:29.530629059Z ovnk_database_store/ovnkube-node-kq6z5_sbdb
2026-05-28T21:30:29.566557365Z ovnk_database_store/ovnkube-node-kq6z5_nbdb
2026-05-28T21:30:29.583132088Z INFO: Network log collection complete.
2026-05-28T21:30:29.715773370Z Wrote inspect data to must-gather.
2026-05-28T21:30:30.083996448Z Wrote inspect data to must-gather.
2026-05-28T21:30:30.368221679Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T21:30:30.371008161Z INFO: Fallback to identify the container image from release info
2026-05-28T21:30:30.576341478Z Gathering data for ns/openshift-network-console...
2026-05-28T21:30:30.910418602Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T21:30:31.052511504Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T21:30:31.112236085Z Gathering data for ns/openshift-console-operator...
2026-05-28T21:30:31.216431833Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T21:30:31.221362672Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T21:30:31.667870145Z Gathering data for ns/openshift-console...
2026-05-28T21:30:32.176942567Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T21:30:32.359331513Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T21:30:32.642734610Z Gathering data for ns/openshift-dns-operator...
2026-05-28T21:30:32.985860656Z Wrote inspect data to must-gather.
2026-05-28T21:30:32.991606469Z Gathering data for ns/openshift-dns...
2026-05-28T21:30:33.481795625Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T21:30:33.639701336Z Gathering data for ns/openshift-image-registry...
2026-05-28T21:30:34.132119613Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T21:30:34.572727318Z Gathering data for ns/openshift-ingress...
2026-05-28T21:30:34.609374206Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T21:30:35.385835847Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T21:30:35.731265227Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T21:30:35.874818777Z Gathering data for ns/openshift-insights...
2026-05-28T21:30:36.226176050Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T21:30:36.840873114Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T21:30:37.829061332Z Gathering data for ns/opendatahub...
2026-05-28T21:30:37.973011205Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T21:30:39.088070933Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T21:30:39.334377315Z Collecting performance related data for node ip-10-0-133-41.ec2.internal
2026-05-28T21:30:39.342623091Z Collecting performance related data for node ip-10-0-140-94.ec2.internal
2026-05-28T21:30:39.350495897Z Collecting performance related data for node ip-10-0-136-77.ec2.internal
2026-05-28T21:30:39.380452414Z Gathering data for ns/openshift-lws-operator...
2026-05-28T21:30:40.874980661Z Gathering data for ns/openshift-config...
2026-05-28T21:30:41.197640187Z Gathering data for ns/openshift-config-managed...
2026-05-28T21:30:41.231872276Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T21:30:41.662787306Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T21:30:42.020238712Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T21:30:42.415421375Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T21:30:42.754495925Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T21:30:43.132375042Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T21:30:43.481334121Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T21:30:43.836565449Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T21:30:44.223041364Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T21:30:44.580918478Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T21:30:45.176929185Z Gathering data for ns/openshift-multus...
2026-05-28T21:30:46.239508301Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T21:30:46.645187731Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T21:30:47.768393230Z Collecting kubelet logs for node ip-10-0-133-41.ec2.internal
2026-05-28T21:30:47.770408600Z Collecting kubelet logs for node ip-10-0-140-94.ec2.internal
2026-05-28T21:30:47.772202327Z Collecting kubelet logs for node ip-10-0-136-77.ec2.internal
2026-05-28T21:30:48.181036767Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T21:30:48.186255944Z INFO: Node performance data collection complete.
2026-05-28T21:30:48.264957350Z Gathering data for ns/openshift-host-network...
2026-05-28T21:30:48.655308228Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T21:30:49.231126151Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T21:30:49.698959194Z Gathering data for ns/openshift-network-operator...
2026-05-28T21:30:50.151134406Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T21:30:50.470361898Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T21:30:50.964910639Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T21:30:51.247107565Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-05-28T21:30:51.288638722Z Gathering data for ns/openshift-apiserver...
2026-05-28T21:30:51.730383655Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T21:30:52.041179182Z Gathering data for ns/openshift-controller-manager...
2026-05-28T21:30:52.420477284Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T21:30:53.085222782Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T21:30:53.413435401Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T21:30:53.743148706Z Gathering data for ns/openshift-service-ca...
2026-05-28T21:30:54.147419726Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T21:30:55.137192854Z Wrote inspect data to must-gather.
2026-05-28T21:30:55.137237638Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T21:30:55.137237638Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T21:30:55.271122451Z error: the server doesn't have a resource type "clusters"
2026-05-28T21:30:55.450345274Z Caches written to disk