2026-05-25T06:40:57.375177573Z [disk usage checker] Started
2026-05-25T06:40:57.378663119Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-25T06:40:57.570296348Z Gathering data for ns/openshift-cluster-version...
2026-05-25T06:40:57.679627379Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T06:40:57.754688210Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T06:40:57.824509557Z Gathering data for ns/default...
2026-05-25T06:40:58.083953127Z Gathering data for ns/openshift...
2026-05-25T06:40:58.453271031Z Gathering data for ns/kube-system...
2026-05-25T06:40:59.043929881Z Gathering data for ns/openshift-etcd...
2026-05-25T06:40:59.305509837Z Wrote inspect data to must-gather.
2026-05-25T06:40:59.305546591Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T06:40:59.305546591Z     namespaces "assisted-installer" not found
2026-05-25T06:40:59.517355868Z Waiting on subprocesses to finish execution.
2026-05-25T06:40:59.525004662Z INFO: Gathering HAProxy config files
2026-05-25T06:40:59.525725544Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-25T06:40:59.532446531Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-25T06:40:59.537826168Z INFO: Collecting host service logs for crio
2026-05-25T06:40:59.538198445Z INFO: Collecting host service logs for kubelet
2026-05-25T06:40:59.538584365Z INFO: Collecting host service logs for rpm-ostreed
2026-05-25T06:40:59.538967973Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-25T06:40:59.539332473Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-25T06:40:59.539793876Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-25T06:40:59.540292126Z INFO: Collecting host service logs for NetworkManager
2026-05-25T06:40:59.540666940Z INFO: Collecting host service logs for openvswitch
2026-05-25T06:40:59.541030722Z INFO: Collecting host service logs for ovs-configuration
2026-05-25T06:40:59.541427154Z INFO: Collecting host service logs for ovsdb-server
2026-05-25T06:40:59.541836446Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-25T06:40:59.542256113Z INFO: Waiting for worker host service log collection to complete ...
2026-05-25T06:40:59.552948283Z INFO: Waiting for node performance related collection to complete ...
2026-05-25T06:40:59.557152711Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-25T06:41:00.183528557Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-tgvp4
2026-05-25T06:41:00.183619600Z insights-runtime-extractor-2vfzj
2026-05-25T06:41:00.183642695Z insights-runtime-extractor-njqk6
2026-05-25T06:41:00.183663309Z insights-runtime-extractor-x55vg
2026-05-25T06:41:00.233220898Z No resources found
2026-05-25T06:41:00.303783032Z error: the server doesn't have a resource type "performanceprofile"
2026-05-25T06:41:00.373851913Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-25T06:41:00.479649379Z No resources found in openshift-etcd namespace.
2026-05-25T06:41:00.707468813Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-25T06:41:00.710371409Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-25T06:41:00.760902769Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-25T06:41:00.766202604Z INFO: "metallb-operator" not detected. Skipping.
2026-05-25T06:41:00.931135242Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-25T06:41:00.931230619Z INFO: on-disk MachineConfig config collection complete.
2026-05-25T06:41:00.967824075Z ERROR: No running kube-apiserver pods found
2026-05-25T06:41:01.041684893Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.124564282Z INFO: Worker host service log collection to complete.
2026-05-25T06:41:01.154785079Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-25T06:41:01.296888907Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.374413370Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-25T06:41:01.374413370Z See 'oc rsync -h' for help and examples
2026-05-25T06:41:01.402582569Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.432149338Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.535904781Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-25T06:41:01.572959880Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.591936469Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T06:41:01.679889776Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.719082237Z No resources found
2026-05-25T06:41:01.734333406Z error: the server doesn't have a resource type "machineconfigs"
2026-05-25T06:41:01.746120318Z Gathering data for ns/openshift-monitoring...
2026-05-25T06:41:01.773220271Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-25T06:41:01.835131304Z INFO: 'previous-logs' folder not found on ip-10-0-130-227.ec2.internal, skipping...
2026-05-25T06:41:01.836223223Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.862054793Z tar: Removing leading `/' from member names
2026-05-25T06:41:01.866587730Z INFO: HAProxy config collection complete.
2026-05-25T06:41:01.962599506Z Wrote inspect data to must-gather.
2026-05-25T06:41:01.972903819Z Wrote inspect data to must-gather.
2026-05-25T06:41:02.052976631Z INFO: Getting rules from prometheus-k8s-0
2026-05-25T06:41:02.086604674Z INFO: 'previous-logs' folder not found on ip-10-0-130-80.ec2.internal, skipping...
2026-05-25T06:41:02.183242055Z Wrote inspect data to must-gather.
2026-05-25T06:41:02.214924951Z Wrote inspect data to must-gather.
2026-05-25T06:41:02.336457411Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-25T06:41:02.339215027Z INFO: 'previous-logs' folder not found on ip-10-0-133-179.ec2.internal, skipping...
2026-05-25T06:41:02.339250311Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-25T06:41:02.339282192Z INFO: Machine Config Daemon termination log collection complete.
2026-05-25T06:41:02.383362056Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-25T06:41:02.438427754Z INFO: Getting status/config from prometheus-k8s-0
2026-05-25T06:41:02.448165055Z Wrote inspect data to must-gather.
2026-05-25T06:41:02.524193496Z Wrote inspect data to must-gather.
2026-05-25T06:41:02.704406320Z Wrote inspect data to must-gather.
2026-05-25T06:41:02.707421761Z INFO: INTERCONNECT MODE
2026-05-25T06:41:02.707473137Z INFO: Gathering ovn-kubernetes DBs
2026-05-25T06:41:02.801451818Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-25T06:41:02.852782191Z INFO: Gathering OVN_Northbound from ovnkube-node-lk229...
2026-05-25T06:41:02.853107387Z INFO: Gathering OVN_Northbound from ovnkube-node-wx58k...
2026-05-25T06:41:02.853384517Z INFO: Gathering OVN_Northbound from ovnkube-node-zshd9...
2026-05-25T06:41:02.853785926Z INFO: Gathering OVN_Southbound from ovnkube-node-lk229...
2026-05-25T06:41:02.854110295Z INFO: Gathering OVN_Southbound from ovnkube-node-wx58k...
2026-05-25T06:41:02.854535661Z INFO: Gathering OVN_Southbound from ovnkube-node-zshd9...
2026-05-25T06:41:02.927470139Z Wrote inspect data to must-gather.
2026-05-25T06:41:03.103452967Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-25T06:41:03.129497833Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T06:41:03.185719848Z tar: Removing leading `/' from member names
2026-05-25T06:41:03.196690893Z tar: Removing leading `/' from member names
2026-05-25T06:41:03.211209490Z tar: Removing leading `/' from member names
2026-05-25T06:41:03.223314429Z tar: Removing leading `/' from member names
2026-05-25T06:41:03.223663311Z tar: Removing leading `/' from member names
2026-05-25T06:41:03.224343069Z tar: Removing leading `/' from member names
2026-05-25T06:41:03.241680193Z Wrote inspect data to must-gather.
2026-05-25T06:41:03.372245492Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-25T06:41:03.468938572Z Wrote inspect data to must-gather.
2026-05-25T06:41:03.599473341Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-25T06:41:03.652971154Z INFO: Waiting for network log collection to complete ...
2026-05-25T06:41:03.653021498Z INFO: Waiting for ovnk database copies to complete ...
2026-05-25T06:41:03.653548507Z INFO: Copying ovnk databases complete.
2026-05-25T06:41:03.655104768Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-05-25T06:41:03.657251682Z ovnk_database_store/
2026-05-25T06:41:03.657272989Z ovnk_database_store/ovnkube-node-lk229_nbdb
2026-05-25T06:41:03.662336923Z W0525 06:41:03.662296    1361 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-25T06:41:03.667819481Z Wrote inspect data to must-gather.
2026-05-25T06:41:03.690152920Z ovnk_database_store/ovnkube-node-zshd9_nbdb
2026-05-25T06:41:03.696374310Z ovnk_database_store/ovnkube-node-lk229_sbdb
2026-05-25T06:41:03.713258912Z ovnk_database_store/ovnkube-node-wx58k_nbdb
2026-05-25T06:41:03.722354255Z ovnk_database_store/ovnkube-node-wx58k_sbdb
2026-05-25T06:41:03.736972828Z ovnk_database_store/ovnkube-node-zshd9_sbdb
2026-05-25T06:41:03.753478889Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T06:41:03.756061264Z INFO: Network log collection complete.
2026-05-25T06:41:03.903425247Z Gathering data for ns/openshift-network-console...
2026-05-25T06:41:03.951699366Z Wrote inspect data to must-gather.
2026-05-25T06:41:03.957801261Z INFO: Getting status from alertmanager-main-0
2026-05-25T06:41:04.183322539Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-25T06:41:04.185793463Z INFO: Fallback to identify the container image from release info
2026-05-25T06:41:04.268230197Z Gathering data for ns/openshift-console-operator...
2026-05-25T06:41:04.577062508Z Gathering data for ns/openshift-console...
2026-05-25T06:41:04.677405378Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-25T06:41:04.791115347Z daemonset.apps/perf-node-gather-daemonset created
2026-05-25T06:41:04.912832039Z Waiting for performance profile collector pods to become ready: 1
2026-05-25T06:41:04.975258449Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-25T06:41:05.285082182Z Gathering data for ns/openshift-dns-operator...
2026-05-25T06:41:05.564757922Z Gathering data for ns/openshift-dns...
2026-05-25T06:41:06.028777799Z Waiting for performance profile collector pods to become ready: 2
2026-05-25T06:41:06.101518912Z Gathering data for ns/openshift-image-registry...
2026-05-25T06:41:06.495959549Z Gathering data for ns/openshift-ingress-operator...
2026-05-25T06:41:06.798100209Z Gathering data for ns/openshift-ingress...
2026-05-25T06:41:07.109693418Z Gathering data for ns/openshift-ingress-canary...
2026-05-25T06:41:07.143765132Z Waiting for performance profile collector pods to become ready: 3
2026-05-25T06:41:07.387988891Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-25T06:41:07.430367337Z Gathering data for ns/openshift-insights...
2026-05-25T06:41:08.242016988Z Waiting for performance profile collector pods to become ready: 4
2026-05-25T06:41:09.066159371Z Gathering data for ns/openshift-config...
2026-05-25T06:41:09.300340721Z Gathering data for ns/openshift-config-managed...
2026-05-25T06:41:09.365923658Z Waiting for performance profile collector pods to become ready: 5
2026-05-25T06:41:09.698890965Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-25T06:41:09.925098708Z Gathering data for ns/openshift-kube-apiserver...
2026-05-25T06:41:10.142221303Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-25T06:41:10.389688303Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-25T06:41:10.484182772Z Waiting for performance profile collector pods to become ready: 6
2026-05-25T06:41:10.638925859Z Gathering data for ns/openshift-kube-scheduler...
2026-05-25T06:41:10.856079173Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-25T06:41:11.098454368Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-25T06:41:11.369746711Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-25T06:41:11.594287686Z Waiting for performance profile collector pods to become ready: 7
2026-05-25T06:41:11.645104232Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-25T06:41:12.076730221Z Gathering data for ns/openshift-multus...
2026-05-25T06:41:12.392347108Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-25T06:41:12.689155335Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-25T06:41:12.931081643Z Collecting performance related data for node ip-10-0-130-227.ec2.internal
2026-05-25T06:41:12.941788588Z Collecting performance related data for node ip-10-0-130-80.ec2.internal
2026-05-25T06:41:12.950671710Z Collecting performance related data for node ip-10-0-133-179.ec2.internal
2026-05-25T06:41:13.396325247Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-25T06:41:14.753103381Z Gathering data for ns/openshift-host-network...
2026-05-25T06:41:15.011170900Z Gathering data for ns/openshift-network-diagnostics...
2026-05-25T06:41:15.548275952Z Gathering data for ns/openshift-network-node-identity...
2026-05-25T06:41:15.960937772Z Gathering data for ns/openshift-network-operator...
2026-05-25T06:41:16.271508461Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-25T06:41:16.501397956Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-25T06:41:16.932010454Z Gathering data for ns/openshift-apiserver-operator...
2026-05-25T06:41:17.210059443Z Gathering data for ns/openshift-apiserver...
2026-05-25T06:41:17.397817370Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-05-25T06:41:17.554858404Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-25T06:41:17.777172350Z Gathering data for ns/openshift-controller-manager...
2026-05-25T06:41:18.013826214Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-25T06:41:18.604491499Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-25T06:41:18.894584785Z Gathering data for ns/openshift-service-ca-operator...
2026-05-25T06:41:19.153834854Z Gathering data for ns/openshift-service-ca...
2026-05-25T06:41:19.453105554Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-25T06:41:20.256110303Z Wrote inspect data to must-gather.
2026-05-25T06:41:20.256292349Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T06:41:20.256292349Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-25T06:41:20.552404567Z Collecting kubelet logs for node ip-10-0-130-227.ec2.internal
2026-05-25T06:41:20.554132770Z Collecting kubelet logs for node ip-10-0-130-80.ec2.internal
2026-05-25T06:41:20.555800231Z Collecting kubelet logs for node ip-10-0-133-179.ec2.internal
2026-05-25T06:41:20.842095721Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-25T06:41:20.846791141Z INFO: Node performance data collection complete.
2026-05-25T06:41:20.967580202Z error: the server doesn't have a resource type "clusters"
2026-05-25T06:41:21.093727859Z Caches written to disk