2026-05-20T16:37:25.198889097Z [disk usage checker] Started
2026-05-20T16:37:25.202166594Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-05-20T16:37:25.583186274Z Gathering data for ns/openshift-cluster-version...
2026-05-20T16:37:25.712154081Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-20T16:37:25.791290664Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T16:37:25.872548421Z Gathering data for ns/default...
2026-05-20T16:37:26.153944095Z Gathering data for ns/openshift...
2026-05-20T16:37:26.554062401Z Gathering data for ns/kube-system...
2026-05-20T16:37:27.312093577Z Gathering data for ns/openshift-etcd...
2026-05-20T16:37:27.639351770Z Wrote inspect data to must-gather.
2026-05-20T16:37:27.639436520Z error: inspection completed with the errors occurred while gathering data:
2026-05-20T16:37:27.639436520Z     namespaces "assisted-installer" not found
2026-05-20T16:37:27.777962656Z Waiting on subprocesses to finish execution.
2026-05-20T16:37:27.783660277Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-20T16:37:27.787489495Z INFO: Gathering HAProxy config files
2026-05-20T16:37:27.790168664Z INFO: Collecting host service logs for crio
2026-05-20T16:37:27.790651535Z INFO: Collecting host service logs for kubelet
2026-05-20T16:37:27.790980969Z INFO: Collecting host service logs for rpm-ostreed
2026-05-20T16:37:27.791438706Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-20T16:37:27.791806079Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-20T16:37:27.792221861Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-20T16:37:27.792566341Z INFO: Collecting host service logs for NetworkManager
2026-05-20T16:37:27.792891353Z INFO: Collecting host service logs for openvswitch
2026-05-20T16:37:27.793241413Z INFO: Collecting host service logs for ovs-configuration
2026-05-20T16:37:27.793586966Z INFO: Collecting host service logs for ovsdb-server
2026-05-20T16:37:27.793923400Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-20T16:37:27.794322650Z INFO: Waiting for worker host service log collection to complete ...
2026-05-20T16:37:27.813557850Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-20T16:37:27.849438121Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-20T16:37:28.056366203Z INFO: Waiting for node performance related collection to complete ...
2026-05-20T16:37:28.095746089Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-h24kw
2026-05-20T16:37:28.095746089Z insights-runtime-extractor-6bzm6
2026-05-20T16:37:28.095746089Z insights-runtime-extractor-fbfcd
2026-05-20T16:37:28.095746089Z insights-runtime-extractor-zbm2j
2026-05-20T16:37:28.269791346Z No resources found in openshift-etcd namespace.
2026-05-20T16:37:28.603355263Z Wrote inspect data to must-gather.
2026-05-20T16:37:28.759169799Z INFO: "metallb-operator" not detected. Skipping.
2026-05-20T16:37:28.783503209Z No resources found
2026-05-20T16:37:28.806492818Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-20T16:37:28.806533973Z INFO: on-disk MachineConfig config collection complete.
2026-05-20T16:37:28.833214940Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-20T16:37:28.993982144Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-20T16:37:29.025295984Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-20T16:37:29.079504308Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-20T16:37:29.079504308Z See 'oc rsync -h' for help and examples
2026-05-20T16:37:29.115550329Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-20T16:37:29.117274231Z INFO: Worker host service log collection to complete.
2026-05-20T16:37:29.131385448Z ERROR: No running kube-apiserver pods found
2026-05-20T16:37:29.215567607Z error: the server doesn't have a resource type "performanceprofile"
2026-05-20T16:37:29.355336142Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-20T16:37:29.396660600Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-20T16:37:29.400623630Z Wrote inspect data to must-gather.
2026-05-20T16:37:29.573222571Z No resources found
2026-05-20T16:37:29.636518406Z Wrote inspect data to must-gather.
2026-05-20T16:37:29.843565590Z Wrote inspect data to must-gather.
2026-05-20T16:37:29.869729306Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T16:37:29.880157668Z Gathering data for ns/openshift-monitoring...
2026-05-20T16:37:29.940274633Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.031754165Z INFO: 'previous-logs' folder not found on ip-10-0-130-228.ec2.internal, skipping...
2026-05-20T16:37:30.141242429Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.154632300Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-20T16:37:30.212129969Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.213599483Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-05-20T16:37:30.222663879Z error: the server doesn't have a resource type "machineconfigs"
2026-05-20T16:37:30.224041432Z tar: Removing leading `/' from member names
2026-05-20T16:37:30.233025934Z INFO: HAProxy config collection complete.
2026-05-20T16:37:30.240235194Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.360483327Z INFO: Getting rules from prometheus-k8s-0
2026-05-20T16:37:30.361179037Z INFO: 'previous-logs' folder not found on ip-10-0-134-233.ec2.internal, skipping...
2026-05-20T16:37:30.468484637Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.495183034Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.639764352Z INFO: 'previous-logs' folder not found on ip-10-0-135-93.ec2.internal, skipping...
2026-05-20T16:37:30.639797945Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-20T16:37:30.639837537Z INFO: Machine Config Daemon termination log collection complete.
2026-05-20T16:37:30.692782991Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.743217833Z Wrote inspect data to must-gather.
2026-05-20T16:37:30.779417030Z INFO: Getting status/config from prometheus-k8s-0
2026-05-20T16:37:30.847776057Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-20T16:37:31.032021560Z INFO: INTERCONNECT MODE
2026-05-20T16:37:31.032047801Z INFO: Gathering ovn-kubernetes DBs
2026-05-20T16:37:31.050141274Z Wrote inspect data to must-gather.
2026-05-20T16:37:31.163298721Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-20T16:37:31.194214747Z INFO: Gathering OVN_Northbound from ovnkube-node-gvwgm...
2026-05-20T16:37:31.194479745Z INFO: Gathering OVN_Northbound from ovnkube-node-hdrhl...
2026-05-20T16:37:31.194719177Z INFO: Gathering OVN_Northbound from ovnkube-node-nmwf4...
2026-05-20T16:37:31.194987353Z INFO: Gathering OVN_Southbound from ovnkube-node-gvwgm...
2026-05-20T16:37:31.195258840Z INFO: Gathering OVN_Southbound from ovnkube-node-hdrhl...
2026-05-20T16:37:31.195553343Z INFO: Gathering OVN_Southbound from ovnkube-node-nmwf4...
2026-05-20T16:37:31.333728659Z Wrote inspect data to must-gather.
2026-05-20T16:37:31.478642065Z tar: Removing leading `/' from member names
2026-05-20T16:37:31.481126497Z tar: Removing leading `/' from member names
2026-05-20T16:37:31.498431361Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-20T16:37:31.521651925Z tar: Removing leading `/' from member names
2026-05-20T16:37:31.522165404Z tar: Removing leading `/' from member names
2026-05-20T16:37:31.548024096Z tar: Removing leading `/' from member names
2026-05-20T16:37:31.549129701Z tar: Removing leading `/' from member names
2026-05-20T16:37:31.562396649Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-20T16:37:31.618999139Z Wrote inspect data to must-gather.
2026-05-20T16:37:31.747187613Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-20T16:37:31.874636867Z Wrote inspect data to must-gather.
2026-05-20T16:37:31.997816150Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-20T16:37:32.047633014Z INFO: Waiting for network log collection to complete ...
2026-05-20T16:37:32.047666591Z INFO: Waiting for ovnk database copies to complete ...
2026-05-20T16:37:32.048062074Z INFO: Copying ovnk databases complete.
2026-05-20T16:37:32.049721376Z 4.4M	must-gather/network_logs/ovnk_database_store
2026-05-20T16:37:32.051906191Z ovnk_database_store/
2026-05-20T16:37:32.051906191Z ovnk_database_store/ovnkube-node-gvwgm_nbdb
2026-05-20T16:37:32.064050613Z ovnk_database_store/ovnkube-node-gvwgm_sbdb
2026-05-20T16:37:32.089799396Z ovnk_database_store/ovnkube-node-nmwf4_nbdb
2026-05-20T16:37:32.100567870Z ovnk_database_store/ovnkube-node-nmwf4_sbdb
2026-05-20T16:37:32.108919984Z W0520 16:37:32.108878    1368 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-20T16:37:32.114524357Z Wrote inspect data to must-gather.
2026-05-20T16:37:32.122145743Z ovnk_database_store/ovnkube-node-hdrhl_sbdb
2026-05-20T16:37:32.153328269Z ovnk_database_store/ovnkube-node-hdrhl_nbdb
2026-05-20T16:37:32.162151504Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-20T16:37:32.170483209Z INFO: Network log collection complete.
2026-05-20T16:37:32.333494690Z Gathering data for ns/openshift-network-console...
2026-05-20T16:37:32.393831605Z INFO: Getting status from alertmanager-main-0
2026-05-20T16:37:32.412470529Z Wrote inspect data to must-gather.
2026-05-20T16:37:32.692415313Z Wrote inspect data to must-gather.
2026-05-20T16:37:32.749755441Z Gathering data for ns/openshift-console-operator...
2026-05-20T16:37:32.942215365Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-20T16:37:32.945263112Z INFO: Fallback to identify the container image from release info
2026-05-20T16:37:33.122819980Z Gathering data for ns/openshift-console...
2026-05-20T16:37:33.528975161Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-20T16:37:33.555590110Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-20T16:37:33.661430176Z daemonset.apps/perf-node-gather-daemonset created
2026-05-20T16:37:33.773115732Z Waiting for performance profile collector pods to become ready: 1
2026-05-20T16:37:33.963792054Z Gathering data for ns/openshift-dns-operator...
2026-05-20T16:37:34.270876405Z Gathering data for ns/openshift-dns...
2026-05-20T16:37:34.881868555Z Gathering data for ns/openshift-image-registry...
2026-05-20T16:37:34.900262135Z Waiting for performance profile collector pods to become ready: 2
2026-05-20T16:37:35.218633932Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-05-20T16:37:35.370273355Z Gathering data for ns/openshift-ingress-operator...
2026-05-20T16:37:35.688993703Z Gathering data for ns/openshift-ingress...
2026-05-20T16:37:36.028622748Z Waiting for performance profile collector pods to become ready: 3
2026-05-20T16:37:36.049082379Z Gathering data for ns/openshift-ingress-canary...
2026-05-20T16:37:36.423521609Z Gathering data for ns/openshift-insights...
2026-05-20T16:37:37.144347197Z Waiting for performance profile collector pods to become ready: 4
2026-05-20T16:37:38.264194857Z Waiting for performance profile collector pods to become ready: 5
2026-05-20T16:37:38.459536896Z Gathering data for ns/kserve...
2026-05-20T16:37:39.398780940Z Waiting for performance profile collector pods to become ready: 6
2026-05-20T16:37:40.223008987Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-05-20T16:37:40.524079895Z Waiting for performance profile collector pods to become ready: 7
2026-05-20T16:37:40.533951286Z Gathering data for ns/openshift-config...
2026-05-20T16:37:40.786524242Z Gathering data for ns/openshift-config-managed...
2026-05-20T16:37:41.207997644Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-20T16:37:41.461615683Z Gathering data for ns/openshift-kube-apiserver...
2026-05-20T16:37:41.643189878Z Waiting for performance profile collector pods to become ready: 8
2026-05-20T16:37:41.734711392Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-20T16:37:41.999908951Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-20T16:37:42.283315889Z Gathering data for ns/openshift-kube-scheduler...
2026-05-20T16:37:42.544785475Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-20T16:37:42.772525070Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-20T16:37:42.852991704Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-20T16:37:43.039848073Z Collecting performance related data for node ip-10-0-130-228.ec2.internal
2026-05-20T16:37:43.046309837Z Collecting performance related data for node ip-10-0-135-93.ec2.internal
2026-05-20T16:37:43.055502017Z Collecting performance related data for node ip-10-0-134-233.ec2.internal
2026-05-20T16:37:43.233619607Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-20T16:37:43.588044504Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-20T16:37:44.234258411Z Gathering data for ns/openshift-multus...
2026-05-20T16:37:45.230574224Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-05-20T16:37:45.699608994Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-20T16:37:47.426687177Z Gathering data for ns/openshift-host-network...
2026-05-20T16:37:47.735710443Z Gathering data for ns/openshift-network-diagnostics...
2026-05-20T16:37:48.315830282Z Gathering data for ns/openshift-network-node-identity...
2026-05-20T16:37:48.733696699Z Gathering data for ns/openshift-network-operator...
2026-05-20T16:37:49.091923404Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-20T16:37:49.345073903Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-20T16:37:49.772517649Z Gathering data for ns/openshift-apiserver-operator...
2026-05-20T16:37:50.063984288Z Gathering data for ns/openshift-apiserver...
2026-05-20T16:37:50.236380588Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-05-20T16:37:50.459320644Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-20T16:37:50.749661424Z Gathering data for ns/openshift-controller-manager...
2026-05-20T16:37:50.943967699Z Collecting kubelet logs for node ip-10-0-130-228.ec2.internal
2026-05-20T16:37:50.946171569Z Collecting kubelet logs for node ip-10-0-135-93.ec2.internal
2026-05-20T16:37:50.948439815Z Collecting kubelet logs for node ip-10-0-134-233.ec2.internal
2026-05-20T16:37:51.045757437Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-20T16:37:51.388652591Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-20T16:37:51.395572855Z INFO: Node performance data collection complete.
2026-05-20T16:37:51.741716554Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-20T16:37:52.033260256Z Gathering data for ns/openshift-service-ca-operator...
2026-05-20T16:37:52.342013166Z Gathering data for ns/openshift-service-ca...
2026-05-20T16:37:52.699444571Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-20T16:37:53.553941824Z Wrote inspect data to must-gather.
2026-05-20T16:37:53.553989238Z error: inspection completed with the errors occurred while gathering data:
2026-05-20T16:37:53.553989238Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-20T16:37:53.683580087Z error: the server doesn't have a resource type "clusters"
2026-05-20T16:37:53.828486691Z Caches written to disk