2026-04-16T17:27:56.256689091Z [disk usage checker] Started
2026-04-16T17:27:56.260033616Z [disk usage checker] Volume usage percentage: current = 40 ; allowed = 70
2026-04-16T17:27:58.686584897Z Gathering data for ns/openshift-cluster-version...
2026-04-16T17:27:58.806070044Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T17:27:58.883997423Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T17:27:58.958698563Z Gathering data for ns/default...
2026-04-16T17:27:59.229631924Z Gathering data for ns/openshift...
2026-04-16T17:27:59.643923501Z Gathering data for ns/kube-system...
2026-04-16T17:28:00.456104071Z Gathering data for ns/openshift-etcd...
2026-04-16T17:28:00.760403259Z Wrote inspect data to must-gather.
2026-04-16T17:28:00.760435489Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T17:28:00.760435489Z     namespaces "assisted-installer" not found
2026-04-16T17:28:00.865133322Z Waiting on subprocesses to finish execution.
2026-04-16T17:28:00.870823328Z INFO: Collecting host service logs for crio
2026-04-16T17:28:00.871321723Z INFO: Collecting host service logs for kubelet
2026-04-16T17:28:00.871631076Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T17:28:00.871967712Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T17:28:00.872300476Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T17:28:00.872649846Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T17:28:00.873015876Z INFO: Collecting host service logs for NetworkManager
2026-04-16T17:28:00.873015876Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T17:28:00.873443134Z INFO: Collecting host service logs for openvswitch
2026-04-16T17:28:00.873784917Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T17:28:00.874111910Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T17:28:00.874437847Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T17:28:00.874837699Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T17:28:00.882587654Z INFO: Gathering HAProxy config files
2026-04-16T17:28:00.884009032Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T17:28:00.901948949Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T17:28:01.243165203Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T17:28:01.243251168Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T17:28:01.382622869Z [disk usage checker] Volume usage percentage: current = 40 ; allowed = 70
2026-04-16T17:28:01.410046293Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T17:28:01.513407647Z No resources found in openshift-etcd namespace.
2026-04-16T17:28:01.525118713Z No resources found
2026-04-16T17:28:01.539616494Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T17:28:01.739708890Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T17:28:01.868165239Z Wrote inspect data to must-gather.
2026-04-16T17:28:01.963748552Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T17:28:02.068528510Z INFO: Worker host service log collection to complete.
2026-04-16T17:28:02.088196188Z INFO: Collecting Insights Archives from insights-operator-5785d4fcdd-qhdbc
2026-04-16T17:28:02.088384287Z insights-runtime-extractor-j72gz
2026-04-16T17:28:02.088411865Z insights-runtime-extractor-jlfdr
2026-04-16T17:28:02.088435247Z insights-runtime-extractor-nknjf
2026-04-16T17:28:02.159331653Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T17:28:02.197022309Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T17:28:02.308005844Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T17:28:02.598751193Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T17:28:02.598751193Z See 'oc rsync -h' for help and examples
2026-04-16T17:28:02.606108473Z Gathering data for ns/openshift-monitoring...
2026-04-16T17:28:02.791901112Z ERROR: No running kube-apiserver pods found
2026-04-16T17:28:02.837048906Z No resources found
2026-04-16T17:28:02.863195357Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T17:28:02.976855289Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T17:28:02.983939372Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T17:28:03.012800535Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.041999289Z INFO: 'previous-logs' folder not found on ip-10-0-132-191.ec2.internal, skipping...
2026-04-16T17:28:03.058638910Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.091452954Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.105362864Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T17:28:03.141515695Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.184413808Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.223733505Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.340208840Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T17:28:03.348662574Z tar: Removing leading `/' from member names
2026-04-16T17:28:03.354026346Z INFO: HAProxy config collection complete.
2026-04-16T17:28:03.400395842Z INFO: 'previous-logs' folder not found on ip-10-0-139-12.ec2.internal, skipping...
2026-04-16T17:28:03.420913697Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.552769882Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T17:28:03.600628884Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.685403055Z INFO: 'previous-logs' folder not found on ip-10-0-140-164.ec2.internal, skipping...
2026-04-16T17:28:03.685437622Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T17:28:03.685463481Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T17:28:03.696833925Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.829147721Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.932876156Z Wrote inspect data to must-gather.
2026-04-16T17:28:03.982714561Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T17:28:04.006360137Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T17:28:04.124292825Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T17:28:04.224247324Z Wrote inspect data to must-gather.
2026-04-16T17:28:04.248256315Z INFO: INTERCONNECT MODE
2026-04-16T17:28:04.248282168Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T17:28:04.406317291Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T17:28:04.425011416Z INFO: Gathering OVN_Northbound from ovnkube-node-b42xz...
2026-04-16T17:28:04.425339460Z INFO: Gathering OVN_Northbound from ovnkube-node-sg9b4...
2026-04-16T17:28:04.425594233Z INFO: Gathering OVN_Northbound from ovnkube-node-wd8ld...
2026-04-16T17:28:04.425868154Z INFO: Gathering OVN_Southbound from ovnkube-node-b42xz...
2026-04-16T17:28:04.426126732Z INFO: Gathering OVN_Southbound from ovnkube-node-sg9b4...
2026-04-16T17:28:04.426368921Z INFO: Gathering OVN_Southbound from ovnkube-node-wd8ld...
2026-04-16T17:28:04.490115595Z Wrote inspect data to must-gather.
2026-04-16T17:28:04.723632461Z tar: Removing leading `/' from member names
2026-04-16T17:28:04.748481799Z tar: Removing leading `/' from member names
2026-04-16T17:28:04.757999787Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T17:28:04.760902531Z tar: Removing leading `/' from member names
2026-04-16T17:28:04.772350273Z tar: Removing leading `/' from member names
2026-04-16T17:28:04.785464296Z tar: Removing leading `/' from member names
2026-04-16T17:28:04.792950620Z tar: Removing leading `/' from member names
2026-04-16T17:28:04.890238030Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T17:28:04.980658965Z Wrote inspect data to must-gather.
2026-04-16T17:28:05.044670084Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T17:28:05.088918784Z Gathering data for ns/openshift-network-console...
2026-04-16T17:28:05.289668106Z INFO: Waiting for network log collection to complete ...
2026-04-16T17:28:05.289701216Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T17:28:05.290183444Z INFO: Copying ovnk databases complete.
2026-04-16T17:28:05.291976961Z 4.4M	must-gather/network_logs/ovnk_database_store
2026-04-16T17:28:05.294263963Z ovnk_database_store/
2026-04-16T17:28:05.294263963Z ovnk_database_store/ovnkube-node-b42xz_nbdb
2026-04-16T17:28:05.307060503Z ovnk_database_store/ovnkube-node-b42xz_sbdb
2026-04-16T17:28:05.312393051Z Wrote inspect data to must-gather.
2026-04-16T17:28:05.330294542Z ovnk_database_store/ovnkube-node-wd8ld_nbdb
2026-04-16T17:28:05.347771201Z ovnk_database_store/ovnkube-node-sg9b4_sbdb
2026-04-16T17:28:05.354058688Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T17:28:05.373060123Z ovnk_database_store/ovnkube-node-wd8ld_sbdb
2026-04-16T17:28:05.407011550Z ovnk_database_store/ovnkube-node-sg9b4_nbdb
2026-04-16T17:28:05.420217462Z INFO: Network log collection complete.
2026-04-16T17:28:05.565041474Z Gathering data for ns/openshift-console-operator...
2026-04-16T17:28:05.630916808Z Wrote inspect data to must-gather.
2026-04-16T17:28:05.790776268Z INFO: Getting status from alertmanager-main-0
2026-04-16T17:28:05.842126523Z W0416 17:28:05.842078    1392 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T17:28:05.938032311Z Wrote inspect data to must-gather.
2026-04-16T17:28:05.948872489Z Gathering data for ns/openshift-console...
2026-04-16T17:28:06.212878468Z Wrote inspect data to must-gather.
2026-04-16T17:28:06.384180088Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T17:28:06.486506148Z [disk usage checker] Volume usage percentage: current = 40 ; allowed = 70
2026-04-16T17:28:06.488022812Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T17:28:06.491006433Z INFO: Fallback to identify the container image from release info
2026-04-16T17:28:06.712479844Z Gathering data for ns/openshift-dns-operator...
2026-04-16T17:28:07.009357003Z Gathering data for ns/openshift-dns...
2026-04-16T17:28:07.064467611Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc77fc5f1546189d6b57967ccc684e794bd846ea36c3e3c1e61d83a684f0f9b9
2026-04-16T17:28:07.174657431Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T17:28:07.308213893Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T17:28:07.578265763Z Gathering data for ns/openshift-image-registry...
2026-04-16T17:28:08.033143568Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T17:28:08.377188066Z Gathering data for ns/openshift-ingress...
2026-04-16T17:28:08.445849854Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T17:28:08.687899780Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T17:28:09.014331565Z Gathering data for ns/openshift-insights...
2026-04-16T17:28:09.573944460Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T17:28:10.709291380Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T17:28:11.117744284Z Gathering data for ns/kserve...
2026-04-16T17:28:11.491044544Z [disk usage checker] Volume usage percentage: current = 40 ; allowed = 70
2026-04-16T17:28:11.820625878Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T17:28:12.940865306Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T17:28:13.108931385Z Gathering data for ns/openshift-config...
2026-04-16T17:28:13.379741703Z Gathering data for ns/openshift-config-managed...
2026-04-16T17:28:13.784451330Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T17:28:14.031865182Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T17:28:14.077739574Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T17:28:14.345583225Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T17:28:14.351470355Z Collecting performance related data for node ip-10-0-139-12.ec2.internal
2026-04-16T17:28:14.358862615Z Collecting performance related data for node ip-10-0-140-164.ec2.internal
2026-04-16T17:28:14.367467147Z Collecting performance related data for node ip-10-0-132-191.ec2.internal
2026-04-16T17:28:14.676983087Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T17:28:15.023249716Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T17:28:15.320218431Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T17:28:15.685183968Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T17:28:16.035437500Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T17:28:16.304288944Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T17:28:16.498395495Z [disk usage checker] Volume usage percentage: current = 40 ; allowed = 70
2026-04-16T17:28:16.775047238Z Gathering data for ns/openshift-multus...
2026-04-16T17:28:18.131541247Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T17:28:19.749179199Z Gathering data for ns/openshift-host-network...
2026-04-16T17:28:20.019453121Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T17:28:20.488240060Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T17:28:20.878085259Z Gathering data for ns/openshift-network-operator...
2026-04-16T17:28:21.225169357Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T17:28:21.502044826Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T17:28:21.503074874Z [disk usage checker] Volume usage percentage: current = 40 ; allowed = 70
2026-04-16T17:28:21.942973161Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T17:28:22.212675105Z Gathering data for ns/openshift-apiserver...
2026-04-16T17:28:22.541405759Z Collecting kubelet logs for node ip-10-0-139-12.ec2.internal
2026-04-16T17:28:22.543219370Z Collecting kubelet logs for node ip-10-0-140-164.ec2.internal
2026-04-16T17:28:22.545308613Z Collecting kubelet logs for node ip-10-0-132-191.ec2.internal
2026-04-16T17:28:22.562311249Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T17:28:22.861696065Z Gathering data for ns/openshift-controller-manager...
2026-04-16T17:28:22.976197692Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T17:28:22.981689733Z INFO: Node performance data collection complete.
2026-04-16T17:28:23.157787328Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T17:28:23.748373068Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T17:28:24.000736875Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T17:28:24.273809130Z Gathering data for ns/openshift-service-ca...
2026-04-16T17:28:24.576162391Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T17:28:25.389454392Z Wrote inspect data to must-gather.
2026-04-16T17:28:25.389497052Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T17:28:25.389497052Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T17:28:25.515370847Z error: the server doesn't have a resource type "clusters"
2026-04-16T17:28:25.676713901Z Caches written to disk