2026-05-21T15:11:56.014241847Z [disk usage checker] Started
2026-05-21T15:11:56.018108678Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T15:11:56.256544118Z Gathering data for ns/openshift-cluster-version...
2026-05-21T15:11:56.419126508Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T15:11:56.531176054Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T15:11:56.642886198Z Gathering data for ns/default...
2026-05-21T15:11:57.048157993Z Gathering data for ns/openshift...
2026-05-21T15:11:57.574464071Z Gathering data for ns/kube-system...
2026-05-21T15:11:58.375016655Z Gathering data for ns/openshift-etcd...
2026-05-21T15:11:58.452042758Z Waiting on subprocesses to finish execution.
2026-05-21T15:11:58.459734086Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-21T15:11:58.462865076Z INFO: Gathering HAProxy config files
2026-05-21T15:11:58.470901171Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-21T15:11:58.476577337Z INFO: Collecting host service logs for crio
2026-05-21T15:11:58.476925832Z INFO: Collecting host service logs for kubelet
2026-05-21T15:11:58.477319866Z INFO: Collecting host service logs for rpm-ostreed
2026-05-21T15:11:58.477678404Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-21T15:11:58.478042645Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-21T15:11:58.478437839Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-21T15:11:58.478815996Z INFO: Collecting host service logs for NetworkManager
2026-05-21T15:11:58.479212476Z INFO: Collecting host service logs for openvswitch
2026-05-21T15:11:58.479562103Z INFO: Collecting host service logs for ovs-configuration
2026-05-21T15:11:58.479932684Z INFO: Collecting host service logs for ovsdb-server
2026-05-21T15:11:58.480318725Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-21T15:11:58.480732637Z INFO: Waiting for worker host service log collection to complete ...
2026-05-21T15:11:58.489274790Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-21T15:11:58.631243102Z INFO: Waiting for node performance related collection to complete ...
2026-05-21T15:11:59.035835522Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-21T15:11:59.095018608Z No resources found in openshift-etcd namespace.
2026-05-21T15:11:59.472085573Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-21T15:11:59.575895418Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-21T15:11:59.593697840Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-2kbbg
2026-05-21T15:11:59.593761976Z insights-runtime-extractor-frdm7
2026-05-21T15:11:59.593785291Z insights-runtime-extractor-mndnd
2026-05-21T15:11:59.593806742Z insights-runtime-extractor-phjqr
2026-05-21T15:11:59.804646303Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-21T15:11:59.804754911Z INFO: on-disk MachineConfig config collection complete.
2026-05-21T15:11:59.824005429Z No resources found
2026-05-21T15:11:59.870816076Z INFO: Worker host service log collection to complete.
2026-05-21T15:11:59.891233644Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-21T15:11:59.966889337Z INFO: "metallb-operator" not detected. Skipping.
2026-05-21T15:11:59.971654315Z error: the server doesn't have a resource type "performanceprofile"
2026-05-21T15:12:00.038541626Z Wrote inspect data to must-gather.
2026-05-21T15:12:00.038582561Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T15:12:00.038582561Z     namespaces "assisted-installer" not found
2026-05-21T15:12:00.327165081Z ERROR: No running kube-apiserver pods found
2026-05-21T15:12:00.343055790Z Wrote inspect data to must-gather.
2026-05-21T15:12:00.425908308Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-21T15:12:00.495914345Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-21T15:12:00.495914345Z See 'oc rsync -h' for help and examples
2026-05-21T15:12:00.552034401Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-21T15:12:00.795733180Z INFO: 'previous-logs' folder not found on ip-10-0-129-1.ec2.internal, skipping...
2026-05-21T15:12:00.987516416Z No resources found
2026-05-21T15:12:01.057307560Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T15:12:01.068875920Z Wrote inspect data to must-gather.
2026-05-21T15:12:01.072379738Z Gathering data for ns/kuadrant-system...
2026-05-21T15:12:01.091152359Z Wrote inspect data to must-gather.
2026-05-21T15:12:01.092045804Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T15:12:01.140816533Z Wrote inspect data to must-gather.
2026-05-21T15:12:01.228950554Z Wrote inspect data to must-gather.
2026-05-21T15:12:01.266457776Z Wrote inspect data to must-gather.
2026-05-21T15:12:01.281293117Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-21T15:12:01.474049218Z tar: Removing leading `/' from member names
2026-05-21T15:12:01.481603696Z INFO: HAProxy config collection complete.
2026-05-21T15:12:01.544207288Z error: the server doesn't have a resource type "machineconfigs"
2026-05-21T15:12:01.581334935Z INFO: 'previous-logs' folder not found on ip-10-0-132-246.ec2.internal, skipping...
2026-05-21T15:12:01.656996850Z Wrote inspect data to must-gather.
2026-05-21T15:12:01.756787649Z INFO: Getting rules from prometheus-k8s-0
2026-05-21T15:12:01.917653357Z Wrote inspect data to must-gather.
2026-05-21T15:12:01.960973035Z INFO: 'previous-logs' folder not found on ip-10-0-140-221.ec2.internal, skipping...
2026-05-21T15:12:01.961096128Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-21T15:12:01.961159894Z INFO: Machine Config Daemon termination log collection complete.
2026-05-21T15:12:02.083418217Z Wrote inspect data to must-gather.
2026-05-21T15:12:02.185976382Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-21T15:12:02.289215446Z Wrote inspect data to must-gather.
2026-05-21T15:12:02.312194420Z INFO: Getting status/config from prometheus-k8s-0
2026-05-21T15:12:02.457727247Z Wrote inspect data to must-gather.
2026-05-21T15:12:02.516830260Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-21T15:12:02.786488167Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-21T15:12:02.855081693Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-21T15:12:02.857053378Z INFO: INTERCONNECT MODE
2026-05-21T15:12:02.857153781Z INFO: Gathering ovn-kubernetes DBs
2026-05-21T15:12:02.931186030Z Wrote inspect data to must-gather.
2026-05-21T15:12:03.044764800Z Gathering data for ns/openshift-monitoring...
2026-05-21T15:12:03.091443854Z INFO: Gathering OVN_Northbound from ovnkube-node-4kj6t...
2026-05-21T15:12:03.091668792Z INFO: Gathering OVN_Northbound from ovnkube-node-64v5f...
2026-05-21T15:12:03.091885907Z INFO: Gathering OVN_Northbound from ovnkube-node-phmc2...
2026-05-21T15:12:03.092183372Z INFO: Gathering OVN_Southbound from ovnkube-node-4kj6t...
2026-05-21T15:12:03.092478203Z INFO: Gathering OVN_Southbound from ovnkube-node-64v5f...
2026-05-21T15:12:03.093144744Z INFO: Gathering OVN_Southbound from ovnkube-node-phmc2...
2026-05-21T15:12:03.317695551Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-21T15:12:03.385678102Z Wrote inspect data to must-gather.
2026-05-21T15:12:03.476896051Z tar: Removing leading `/' from member names
2026-05-21T15:12:03.491351539Z tar: Removing leading `/' from member names
2026-05-21T15:12:03.516940265Z tar: Removing leading `/' from member names
2026-05-21T15:12:03.516940265Z tar: Removing leading `/' from member names
2026-05-21T15:12:03.556480507Z tar: Removing leading `/' from member names
2026-05-21T15:12:03.582820064Z tar: Removing leading `/' from member names
2026-05-21T15:12:03.667632650Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-21T15:12:03.856001677Z Wrote inspect data to must-gather.
2026-05-21T15:12:03.969097415Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-21T15:12:04.206897669Z INFO: Waiting for network log collection to complete ...
2026-05-21T15:12:04.206943534Z INFO: Waiting for ovnk database copies to complete ...
2026-05-21T15:12:04.207358651Z INFO: Copying ovnk databases complete.
2026-05-21T15:12:04.208954740Z 4.1M	must-gather/network_logs/ovnk_database_store
2026-05-21T15:12:04.211586598Z ovnk_database_store/
2026-05-21T15:12:04.211586598Z ovnk_database_store/ovnkube-node-phmc2_sbdb
2026-05-21T15:12:04.219328316Z Wrote inspect data to must-gather.
2026-05-21T15:12:04.236151915Z ovnk_database_store/ovnkube-node-phmc2_nbdb
2026-05-21T15:12:04.246683806Z ovnk_database_store/ovnkube-node-4kj6t_sbdb
2026-05-21T15:12:04.271121118Z ovnk_database_store/ovnkube-node-4kj6t_nbdb
2026-05-21T15:12:04.281690550Z ovnk_database_store/ovnkube-node-64v5f_nbdb
2026-05-21T15:12:04.292957082Z ovnk_database_store/ovnkube-node-64v5f_sbdb
2026-05-21T15:12:04.324852818Z INFO: Network log collection complete.
2026-05-21T15:12:04.418197946Z INFO: Getting status from alertmanager-main-0
2026-05-21T15:12:04.585663560Z Wrote inspect data to must-gather.
2026-05-21T15:12:04.920760050Z Wrote inspect data to must-gather.
2026-05-21T15:12:05.310745798Z Wrote inspect data to must-gather.
2026-05-21T15:12:05.427508732Z Gathering data for ns/openshift-network-console...
2026-05-21T15:12:05.598567612Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-21T15:12:05.604427774Z INFO: Fallback to identify the container image from release info
2026-05-21T15:12:05.953432778Z Gathering data for ns/openshift-console-operator...
2026-05-21T15:12:06.117464501Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T15:12:06.218492902Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-21T15:12:06.374614840Z daemonset.apps/perf-node-gather-daemonset created
2026-05-21T15:12:06.453338082Z Gathering data for ns/openshift-console...
2026-05-21T15:12:06.523689899Z Waiting for performance profile collector pods to become ready: 1
2026-05-21T15:12:07.016803433Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-21T15:12:07.453208295Z Gathering data for ns/openshift-dns-operator...
2026-05-21T15:12:07.661025423Z Waiting for performance profile collector pods to become ready: 2
2026-05-21T15:12:07.831373826Z Gathering data for ns/openshift-dns...
2026-05-21T15:12:08.236050061Z Wrote inspect data to must-gather.
2026-05-21T15:12:08.486569378Z Gathering data for ns/openshift-image-registry...
2026-05-21T15:12:08.778639377Z Waiting for performance profile collector pods to become ready: 3
2026-05-21T15:12:08.973629988Z Gathering data for ns/openshift-ingress-operator...
2026-05-21T15:12:09.320818897Z Gathering data for ns/openshift-ingress...
2026-05-21T15:12:09.895556362Z Waiting for performance profile collector pods to become ready: 4
2026-05-21T15:12:10.204725379Z Gathering data for ns/openshift-ingress-canary...
2026-05-21T15:12:10.663430591Z Gathering data for ns/openshift-insights...
2026-05-21T15:12:11.031085262Z Waiting for performance profile collector pods to become ready: 5
2026-05-21T15:12:11.122011455Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-21T15:12:12.142093282Z Waiting for performance profile collector pods to become ready: 6
2026-05-21T15:12:12.621488452Z Gathering data for ns/opendatahub...
2026-05-21T15:12:13.264214021Z Waiting for performance profile collector pods to become ready: 7
2026-05-21T15:12:14.149246032Z Gathering data for ns/openshift-lws-operator...
2026-05-21T15:12:14.379160371Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-21T15:12:14.648464783Z Collecting performance related data for node ip-10-0-129-1.ec2.internal
2026-05-21T15:12:14.657350123Z Collecting performance related data for node ip-10-0-132-246.ec2.internal
2026-05-21T15:12:14.676030629Z Collecting performance related data for node ip-10-0-140-221.ec2.internal
2026-05-21T15:12:15.544246697Z Gathering data for ns/openshift-config...
2026-05-21T15:12:15.925524676Z Gathering data for ns/openshift-config-managed...
2026-05-21T15:12:16.128607401Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T15:12:16.464311206Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-21T15:12:16.753430270Z Gathering data for ns/openshift-kube-apiserver...
2026-05-21T15:12:17.089002232Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-21T15:12:17.556543190Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-21T15:12:17.929197950Z Gathering data for ns/openshift-kube-scheduler...
2026-05-21T15:12:18.255436875Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-21T15:12:18.586111987Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-21T15:12:18.950304850Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-21T15:12:19.308226590Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-21T15:12:19.985138712Z Gathering data for ns/openshift-multus...
2026-05-21T15:12:21.136668873Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T15:12:21.476541085Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-21T15:12:23.086215247Z Gathering data for ns/openshift-host-network...
2026-05-21T15:12:23.274445921Z Collecting kubelet logs for node ip-10-0-129-1.ec2.internal
2026-05-21T15:12:23.276192700Z Collecting kubelet logs for node ip-10-0-132-246.ec2.internal
2026-05-21T15:12:23.277909087Z Collecting kubelet logs for node ip-10-0-140-221.ec2.internal
2026-05-21T15:12:23.541546501Z Gathering data for ns/openshift-network-diagnostics...
2026-05-21T15:12:23.645000586Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-21T15:12:23.650633768Z INFO: Node performance data collection complete.
2026-05-21T15:12:24.148317274Z Gathering data for ns/openshift-network-node-identity...
2026-05-21T15:12:24.607206287Z Gathering data for ns/openshift-network-operator...
2026-05-21T15:12:25.002776628Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-21T15:12:25.325583797Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-21T15:12:25.812891485Z Gathering data for ns/openshift-apiserver-operator...
2026-05-21T15:12:26.143445257Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-21T15:12:26.155517630Z Gathering data for ns/openshift-apiserver...
2026-05-21T15:12:26.583479021Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-21T15:12:26.897570055Z Gathering data for ns/openshift-controller-manager...
2026-05-21T15:12:27.223632606Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-21T15:12:27.951329985Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-21T15:12:28.279601107Z Gathering data for ns/openshift-service-ca-operator...
2026-05-21T15:12:28.666127370Z Gathering data for ns/openshift-service-ca...
2026-05-21T15:12:29.118492042Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-21T15:12:30.101350479Z Wrote inspect data to must-gather.
2026-05-21T15:12:30.101397500Z error: inspection completed with the errors occurred while gathering data:
2026-05-21T15:12:30.101397500Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-21T15:12:30.238222988Z error: the server doesn't have a resource type "clusters"
2026-05-21T15:12:30.454162174Z Caches written to disk