2026-04-16T21:12:01.636817168Z [disk usage checker] Started
2026-04-16T21:12:01.676958908Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-16T21:12:01.961309616Z Gathering data for ns/openshift-cluster-version...
2026-04-16T21:12:02.091342655Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T21:12:02.187260858Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T21:12:02.268657693Z Gathering data for ns/default...
2026-04-16T21:12:02.569012960Z Gathering data for ns/openshift...
2026-04-16T21:12:03.003935938Z Gathering data for ns/kube-system...
2026-04-16T21:12:03.746583368Z Gathering data for ns/openshift-etcd...
2026-04-16T21:12:04.084823763Z Wrote inspect data to must-gather.
2026-04-16T21:12:04.084945420Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T21:12:04.084945420Z     namespaces "assisted-installer" not found
2026-04-16T21:12:04.212208091Z Waiting on subprocesses to finish execution.
2026-04-16T21:12:04.219879645Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T21:12:04.219879645Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T21:12:04.230132730Z INFO: Collecting host service logs for crio
2026-04-16T21:12:04.230470397Z INFO: Collecting host service logs for kubelet
2026-04-16T21:12:04.230864152Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T21:12:04.231200643Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T21:12:04.231529718Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T21:12:04.231867998Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T21:12:04.232195759Z INFO: Collecting host service logs for NetworkManager
2026-04-16T21:12:04.232535794Z INFO: Collecting host service logs for openvswitch
2026-04-16T21:12:04.232889944Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T21:12:04.233231443Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T21:12:04.233582950Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T21:12:04.233976987Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T21:12:04.234300997Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T21:12:04.270253704Z INFO: Gathering HAProxy config files
2026-04-16T21:12:04.555461088Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T21:12:04.570873620Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T21:12:04.587815197Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T21:12:04.691541512Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T21:12:04.691632660Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T21:12:04.814173756Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T21:12:04.909943226Z Wrote inspect data to must-gather.
2026-04-16T21:12:05.069233044Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T21:12:05.204130591Z No resources found
2026-04-16T21:12:05.347446559Z INFO: Worker host service log collection to complete.
2026-04-16T21:12:05.396025917Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T21:12:05.432415794Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T21:12:05.460637174Z No resources found in openshift-etcd namespace.
2026-04-16T21:12:05.487311088Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T21:12:05.597243234Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-q5swj
2026-04-16T21:12:05.597296740Z insights-runtime-extractor-5rcs6
2026-04-16T21:12:05.597318304Z insights-runtime-extractor-m64qv
2026-04-16T21:12:05.597339776Z insights-runtime-extractor-x9xwk
2026-04-16T21:12:05.743769568Z ERROR: No running kube-apiserver pods found
2026-04-16T21:12:05.987379707Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.066728805Z tar: Removing leading `/' from member names
2026-04-16T21:12:06.079075779Z INFO: HAProxy config collection complete.
2026-04-16T21:12:06.137018292Z No resources found
2026-04-16T21:12:06.148081341Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T21:12:06.164595679Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T21:12:06.304691057Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.331825378Z INFO: 'previous-logs' folder not found on ip-10-0-137-142.ec2.internal, skipping...
2026-04-16T21:12:06.342273350Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T21:12:06.350154673Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.382969905Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.409495519Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T21:12:06.409495519Z See 'oc rsync -h' for help and examples
2026-04-16T21:12:06.446260627Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.488303723Z Gathering data for ns/openshift-monitoring...
2026-04-16T21:12:06.514736097Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.579219831Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T21:12:06.660297720Z INFO: 'previous-logs' folder not found on ip-10-0-138-118.ec2.internal, skipping...
2026-04-16T21:12:06.682888865Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-16T21:12:06.717736942Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.800794056Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T21:12:06.819864907Z Wrote inspect data to must-gather.
2026-04-16T21:12:06.929670535Z INFO: 'previous-logs' folder not found on ip-10-0-142-60.ec2.internal, skipping...
2026-04-16T21:12:06.929704162Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T21:12:06.929712278Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T21:12:06.985696228Z Wrote inspect data to must-gather.
2026-04-16T21:12:07.040632288Z Wrote inspect data to must-gather.
2026-04-16T21:12:07.193971838Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T21:12:07.225578702Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T21:12:07.236983616Z Wrote inspect data to must-gather.
2026-04-16T21:12:07.416141992Z Wrote inspect data to must-gather.
2026-04-16T21:12:07.535292027Z INFO: INTERCONNECT MODE
2026-04-16T21:12:07.535326239Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T21:12:07.626531103Z Wrote inspect data to must-gather.
2026-04-16T21:12:07.632161055Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T21:12:07.695447303Z INFO: Gathering OVN_Northbound from ovnkube-node-8lrq8...
2026-04-16T21:12:07.695726740Z INFO: Gathering OVN_Northbound from ovnkube-node-mgjgp...
2026-04-16T21:12:07.695982148Z INFO: Gathering OVN_Northbound from ovnkube-node-qjb9s...
2026-04-16T21:12:07.696249281Z INFO: Gathering OVN_Southbound from ovnkube-node-8lrq8...
2026-04-16T21:12:07.696480319Z INFO: Gathering OVN_Southbound from ovnkube-node-mgjgp...
2026-04-16T21:12:07.696784396Z INFO: Gathering OVN_Southbound from ovnkube-node-qjb9s...
2026-04-16T21:12:07.954150748Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T21:12:07.964800974Z Wrote inspect data to must-gather.
2026-04-16T21:12:07.978863676Z tar: Removing leading `/' from member names
2026-04-16T21:12:08.011067327Z tar: Removing leading `/' from member names
2026-04-16T21:12:08.013202047Z tar: Removing leading `/' from member names
2026-04-16T21:12:08.017999694Z tar: Removing leading `/' from member names
2026-04-16T21:12:08.029738404Z tar: Removing leading `/' from member names
2026-04-16T21:12:08.030742369Z tar: Removing leading `/' from member names
2026-04-16T21:12:08.052503513Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T21:12:08.210434580Z Wrote inspect data to must-gather.
2026-04-16T21:12:08.210434580Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T21:12:08.450182937Z Wrote inspect data to must-gather.
2026-04-16T21:12:08.481622183Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T21:12:08.527590215Z INFO: Waiting for network log collection to complete ...
2026-04-16T21:12:08.527646715Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T21:12:08.528131317Z INFO: Copying ovnk databases complete.
2026-04-16T21:12:08.530109996Z 4.3M	must-gather/network_logs/ovnk_database_store
2026-04-16T21:12:08.532752555Z ovnk_database_store/
2026-04-16T21:12:08.532752555Z ovnk_database_store/ovnkube-node-mgjgp_sbdb
2026-04-16T21:12:08.560898016Z ovnk_database_store/ovnkube-node-mgjgp_nbdb
2026-04-16T21:12:08.570034193Z ovnk_database_store/ovnkube-node-8lrq8_sbdb
2026-04-16T21:12:08.592544986Z ovnk_database_store/ovnkube-node-qjb9s_sbdb
2026-04-16T21:12:08.624337046Z ovnk_database_store/ovnkube-node-qjb9s_nbdb
2026-04-16T21:12:08.639699107Z ovnk_database_store/ovnkube-node-8lrq8_nbdb
2026-04-16T21:12:08.656496343Z INFO: Network log collection complete.
2026-04-16T21:12:08.666903521Z W0416 21:12:08.666863    1370 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T21:12:08.673512865Z Wrote inspect data to must-gather.
2026-04-16T21:12:08.680292871Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T21:12:08.851837710Z Gathering data for ns/openshift-network-console...
2026-04-16T21:12:08.891200657Z INFO: Getting status from alertmanager-main-0
2026-04-16T21:12:08.980089526Z Wrote inspect data to must-gather.
2026-04-16T21:12:09.241194058Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T21:12:09.244147613Z INFO: Fallback to identify the container image from release info
2026-04-16T21:12:09.253965295Z Gathering data for ns/openshift-console-operator...
2026-04-16T21:12:09.604367279Z Gathering data for ns/openshift-console...
2026-04-16T21:12:09.933544520Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-16T21:12:10.017207776Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T21:12:10.057947339Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T21:12:10.190851798Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T21:12:10.362946217Z Gathering data for ns/openshift-dns-operator...
2026-04-16T21:12:10.669459613Z Gathering data for ns/openshift-dns...
2026-04-16T21:12:11.323376284Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T21:12:11.342364202Z Gathering data for ns/openshift-image-registry...
2026-04-16T21:12:11.688046493Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-16T21:12:11.806901403Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T21:12:12.162194506Z Gathering data for ns/openshift-ingress...
2026-04-16T21:12:12.457117540Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T21:12:12.500079928Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T21:12:12.875987774Z Gathering data for ns/openshift-insights...
2026-04-16T21:12:13.585294607Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T21:12:14.707445350Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T21:12:14.947379825Z Gathering data for ns/kserve...
2026-04-16T21:12:15.848608060Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T21:12:16.688772476Z Gathering data for ns/openshift-config...
2026-04-16T21:12:16.694868695Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-16T21:12:16.966596246Z Gathering data for ns/openshift-config-managed...
2026-04-16T21:12:16.976531539Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T21:12:17.389618226Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T21:12:17.637429717Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T21:12:17.913413812Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T21:12:18.087956523Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T21:12:18.254740331Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T21:12:18.360186199Z Collecting performance related data for node ip-10-0-137-142.ec2.internal
2026-04-16T21:12:18.366858087Z Collecting performance related data for node ip-10-0-138-118.ec2.internal
2026-04-16T21:12:18.374120609Z Collecting performance related data for node ip-10-0-142-60.ec2.internal
2026-04-16T21:12:18.598722610Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T21:12:18.936995333Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T21:12:19.346986048Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T21:12:19.750297321Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T21:12:20.107736971Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T21:12:20.650311349Z Gathering data for ns/openshift-multus...
2026-04-16T21:12:21.701260819Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-16T21:12:22.115628662Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T21:12:23.705398839Z Gathering data for ns/openshift-host-network...
2026-04-16T21:12:24.065491503Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T21:12:24.582521568Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T21:12:24.963548954Z Gathering data for ns/openshift-network-operator...
2026-04-16T21:12:25.335305548Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T21:12:25.635827366Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T21:12:26.077713705Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T21:12:26.415407481Z Gathering data for ns/openshift-apiserver...
2026-04-16T21:12:26.541590756Z Collecting kubelet logs for node ip-10-0-137-142.ec2.internal
2026-04-16T21:12:26.543932027Z Collecting kubelet logs for node ip-10-0-138-118.ec2.internal
2026-04-16T21:12:26.547333104Z Collecting kubelet logs for node ip-10-0-142-60.ec2.internal
2026-04-16T21:12:26.708262824Z [disk usage checker] Volume usage percentage: current = 37 ; allowed = 70
2026-04-16T21:12:26.866821684Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T21:12:26.977295081Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T21:12:26.982825572Z INFO: Node performance data collection complete.
2026-04-16T21:12:27.153453804Z Gathering data for ns/openshift-controller-manager...
2026-04-16T21:12:27.458732125Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T21:12:28.087635091Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T21:12:28.408480810Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T21:12:28.703621628Z Gathering data for ns/openshift-service-ca...
2026-04-16T21:12:29.056714213Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T21:12:29.917722261Z Wrote inspect data to must-gather.
2026-04-16T21:12:29.917763754Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T21:12:29.917763754Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T21:12:30.044116837Z error: the server doesn't have a resource type "clusters"
2026-04-16T21:12:30.208051427Z Caches written to disk