2026-05-28T17:51:11.912460142Z [disk usage checker] Started
2026-05-28T17:51:11.915999778Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-28T17:51:12.151956485Z Gathering data for ns/openshift-cluster-version...
2026-05-28T17:51:12.301797806Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T17:51:12.390936493Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T17:51:12.487998288Z Gathering data for ns/default...
2026-05-28T17:51:12.792886746Z Gathering data for ns/openshift...
2026-05-28T17:51:13.219337954Z Gathering data for ns/kube-system...
2026-05-28T17:51:13.891491028Z Gathering data for ns/openshift-etcd...
2026-05-28T17:51:14.223484573Z Wrote inspect data to must-gather.
2026-05-28T17:51:14.223516401Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T17:51:14.223516401Z     namespaces "assisted-installer" not found
2026-05-28T17:51:14.387726299Z Waiting on subprocesses to finish execution.
2026-05-28T17:51:14.394906881Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T17:51:14.400789762Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T17:51:14.401771837Z INFO: Collecting host service logs for crio
2026-05-28T17:51:14.402124381Z INFO: Collecting host service logs for kubelet
2026-05-28T17:51:14.402268251Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T17:51:14.402514017Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T17:51:14.402843640Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T17:51:14.403195335Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T17:51:14.403517081Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T17:51:14.403828301Z INFO: Collecting host service logs for NetworkManager
2026-05-28T17:51:14.404168152Z INFO: Collecting host service logs for openvswitch
2026-05-28T17:51:14.404497803Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T17:51:14.404819656Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T17:51:14.405160978Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T17:51:14.405548015Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T17:51:14.412278426Z INFO: Gathering HAProxy config files
2026-05-28T17:51:14.605044970Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T17:51:15.082213891Z No resources found in openshift-etcd namespace.
2026-05-28T17:51:15.243702260Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-972s5
2026-05-28T17:51:15.243702260Z insights-runtime-extractor-7968r
2026-05-28T17:51:15.243702260Z insights-runtime-extractor-8tpt5
2026-05-28T17:51:15.243702260Z insights-runtime-extractor-lq6wn
2026-05-28T17:51:15.440450365Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T17:51:15.465335675Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T17:51:15.465460619Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T17:51:15.500112809Z No resources found
2026-05-28T17:51:15.676045821Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T17:51:15.767391220Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T17:51:15.832826978Z Wrote inspect data to must-gather.
2026-05-28T17:51:15.916068443Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T17:51:16.014429624Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T17:51:16.249054346Z INFO: Worker host service log collection to complete.
2026-05-28T17:51:16.369766815Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T17:51:16.396944342Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T17:51:16.512467157Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T17:51:16.512467157Z See 'oc rsync -h' for help and examples
2026-05-28T17:51:16.623305719Z No resources found
2026-05-28T17:51:16.788029451Z Wrote inspect data to must-gather.
2026-05-28T17:51:16.837454179Z Wrote inspect data to must-gather.
2026-05-28T17:51:16.868305059Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T17:51:16.880885331Z Wrote inspect data to must-gather.
2026-05-28T17:51:16.925555595Z ERROR: No running kube-apiserver pods found
2026-05-28T17:51:16.933147631Z Wrote inspect data to must-gather.
2026-05-28T17:51:16.945301706Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-28T17:51:16.987580029Z Gathering data for ns/kuadrant-system...
2026-05-28T17:51:17.021820432Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T17:51:17.034304566Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T17:51:17.051725290Z Wrote inspect data to must-gather.
2026-05-28T17:51:17.055027568Z INFO: 'previous-logs' folder not found on ip-10-0-128-101.ec2.internal, skipping...
2026-05-28T17:51:17.089730340Z tar: Removing leading `/' from member names
2026-05-28T17:51:17.095246185Z INFO: HAProxy config collection complete.
2026-05-28T17:51:17.306983201Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T17:51:17.355157747Z Wrote inspect data to must-gather.
2026-05-28T17:51:17.531312281Z INFO: 'previous-logs' folder not found on ip-10-0-132-206.ec2.internal, skipping...
2026-05-28T17:51:17.750197457Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T17:51:17.769471654Z Wrote inspect data to must-gather.
2026-05-28T17:51:17.850273389Z Wrote inspect data to must-gather.
2026-05-28T17:51:18.015294139Z INFO: 'previous-logs' folder not found on ip-10-0-138-128.ec2.internal, skipping...
2026-05-28T17:51:18.015331373Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T17:51:18.015359384Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T17:51:18.180813606Z Wrote inspect data to must-gather.
2026-05-28T17:51:18.225727388Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T17:51:18.290349213Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T17:51:18.292453999Z Wrote inspect data to must-gather.
2026-05-28T17:51:18.376909748Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T17:51:18.653464731Z W0528 17:51:18.653413    1126 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-28T17:51:18.681039299Z Wrote inspect data to must-gather.
2026-05-28T17:51:18.686493851Z INFO: INTERCONNECT MODE
2026-05-28T17:51:18.686523933Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T17:51:18.799963712Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T17:51:18.817366380Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T17:51:18.912636922Z INFO: Gathering OVN_Northbound from ovnkube-node-mq42g...
2026-05-28T17:51:18.912996394Z INFO: Gathering OVN_Northbound from ovnkube-node-p82jb...
2026-05-28T17:51:18.913286472Z INFO: Gathering OVN_Northbound from ovnkube-node-s5mhs...
2026-05-28T17:51:18.913602625Z INFO: Gathering OVN_Southbound from ovnkube-node-mq42g...
2026-05-28T17:51:18.913871505Z INFO: Gathering OVN_Southbound from ovnkube-node-p82jb...
2026-05-28T17:51:18.914181770Z INFO: Gathering OVN_Southbound from ovnkube-node-s5mhs...
2026-05-28T17:51:19.140111986Z Gathering data for ns/openshift-monitoring...
2026-05-28T17:51:19.161643192Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T17:51:19.200506429Z Wrote inspect data to must-gather.
2026-05-28T17:51:19.273431244Z tar: Removing leading `/' from member names
2026-05-28T17:51:19.301549343Z tar: Removing leading `/' from member names
2026-05-28T17:51:19.365245908Z tar: Removing leading `/' from member names
2026-05-28T17:51:19.365403545Z tar: Removing leading `/' from member names
2026-05-28T17:51:19.373750716Z tar: Removing leading `/' from member names
2026-05-28T17:51:19.417266127Z tar: Removing leading `/' from member names
2026-05-28T17:51:19.560437510Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T17:51:19.682368025Z Wrote inspect data to must-gather.
2026-05-28T17:51:19.832220726Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T17:51:20.025677400Z INFO: Waiting for network log collection to complete ...
2026-05-28T17:51:20.025730193Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T17:51:20.026385909Z INFO: Copying ovnk databases complete.
2026-05-28T17:51:20.028720980Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-05-28T17:51:20.031025388Z ovnk_database_store/
2026-05-28T17:51:20.031025388Z ovnk_database_store/ovnkube-node-mq42g_nbdb
2026-05-28T17:51:20.043895385Z ovnk_database_store/ovnkube-node-mq42g_sbdb
2026-05-28T17:51:20.057305782Z Wrote inspect data to must-gather.
2026-05-28T17:51:20.073827325Z ovnk_database_store/ovnkube-node-s5mhs_sbdb
2026-05-28T17:51:20.100406520Z ovnk_database_store/ovnkube-node-s5mhs_nbdb
2026-05-28T17:51:20.111736086Z ovnk_database_store/ovnkube-node-p82jb_sbdb
2026-05-28T17:51:20.135403658Z ovnk_database_store/ovnkube-node-p82jb_nbdb
2026-05-28T17:51:20.151190198Z INFO: Network log collection complete.
2026-05-28T17:51:20.300076616Z INFO: Getting status from alertmanager-main-0
2026-05-28T17:51:20.418655873Z Wrote inspect data to must-gather.
2026-05-28T17:51:20.744827834Z Wrote inspect data to must-gather.
2026-05-28T17:51:21.046216784Z W0528 17:51:21.046169    1429 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-28T17:51:21.057610293Z Wrote inspect data to must-gather.
2026-05-28T17:51:21.349135292Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T17:51:21.351877986Z INFO: Fallback to identify the container image from release info
2026-05-28T17:51:21.374346751Z Gathering data for ns/openshift-network-console...
2026-05-28T17:51:21.831816052Z Gathering data for ns/openshift-console-operator...
2026-05-28T17:51:21.970176853Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-28T17:51:21.972307248Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T17:51:22.082288418Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T17:51:22.249983210Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T17:51:22.294489678Z Gathering data for ns/openshift-console...
2026-05-28T17:51:22.797188393Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T17:51:23.206681405Z Gathering data for ns/openshift-dns-operator...
2026-05-28T17:51:23.392316144Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T17:51:23.571136936Z Gathering data for ns/openshift-dns...
2026-05-28T17:51:24.054740331Z Wrote inspect data to must-gather.
2026-05-28T17:51:24.184779863Z Gathering data for ns/openshift-image-registry...
2026-05-28T17:51:24.510220914Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T17:51:24.650253950Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T17:51:25.003664953Z Gathering data for ns/openshift-ingress...
2026-05-28T17:51:25.639156784Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T17:51:25.807637489Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T17:51:26.265741905Z Gathering data for ns/openshift-insights...
2026-05-28T17:51:26.764363457Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T17:51:26.975170992Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-28T17:51:27.883342428Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T17:51:28.166084942Z Gathering data for ns/opendatahub...
2026-05-28T17:51:29.012516023Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T17:51:29.587977239Z Gathering data for ns/openshift-lws-operator...
2026-05-28T17:51:30.125404971Z Waiting for performance profile collector pods to become ready: 8
2026-05-28T17:51:30.764507297Z Gathering data for ns/openshift-config...
2026-05-28T17:51:31.049875837Z Gathering data for ns/openshift-config-managed...
2026-05-28T17:51:31.243094312Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T17:51:31.496722459Z Collecting performance related data for node ip-10-0-138-128.ec2.internal
2026-05-28T17:51:31.504155989Z Collecting performance related data for node ip-10-0-132-206.ec2.internal
2026-05-28T17:51:31.518284109Z Collecting performance related data for node ip-10-0-128-101.ec2.internal
2026-05-28T17:51:31.526091975Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T17:51:31.925228787Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T17:51:31.983963845Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-28T17:51:32.314986018Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T17:51:32.711315183Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T17:51:33.125140137Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T17:51:33.433004546Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T17:51:33.795157355Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T17:51:34.151876831Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T17:51:34.585459188Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T17:51:35.134248135Z Gathering data for ns/openshift-multus...
2026-05-28T17:51:36.467505041Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T17:51:36.988702306Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-28T17:51:37.954184717Z Gathering data for ns/openshift-host-network...
2026-05-28T17:51:38.266124502Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T17:51:38.797800504Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T17:51:39.224365385Z Gathering data for ns/openshift-network-operator...
2026-05-28T17:51:39.614525416Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T17:51:39.913746243Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T17:51:40.080957693Z Collecting kubelet logs for node ip-10-0-138-128.ec2.internal
2026-05-28T17:51:40.082763380Z Collecting kubelet logs for node ip-10-0-132-206.ec2.internal
2026-05-28T17:51:40.084569912Z Collecting kubelet logs for node ip-10-0-128-101.ec2.internal
2026-05-28T17:51:40.463893947Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T17:51:40.466088449Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T17:51:40.470391670Z INFO: Node performance data collection complete.
2026-05-28T17:51:40.805626455Z Gathering data for ns/openshift-apiserver...
2026-05-28T17:51:41.243856578Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T17:51:41.518555960Z Gathering data for ns/openshift-controller-manager...
2026-05-28T17:51:41.826536561Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T17:51:41.993129988Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-05-28T17:51:42.468146199Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T17:51:42.845168869Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T17:51:43.158892198Z Gathering data for ns/openshift-service-ca...
2026-05-28T17:51:43.521063480Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T17:51:44.464198014Z Wrote inspect data to must-gather.
2026-05-28T17:51:44.464241153Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T17:51:44.464241153Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T17:51:44.597762630Z error: the server doesn't have a resource type "clusters"
2026-05-28T17:51:44.769558884Z Caches written to disk