2026-04-18T03:21:36.790126557Z [disk usage checker] Started
2026-04-18T03:21:36.826126458Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-18T03:21:37.014200786Z Gathering data for ns/openshift-cluster-version...
2026-04-18T03:21:37.174021419Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-18T03:21:37.264191618Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-18T03:21:37.361719048Z Gathering data for ns/default...
2026-04-18T03:21:37.729893971Z Gathering data for ns/openshift...
2026-04-18T03:21:38.199262071Z Gathering data for ns/kube-system...
2026-04-18T03:21:39.029704388Z Gathering data for ns/openshift-etcd...
2026-04-18T03:21:39.328347066Z Waiting on subprocesses to finish execution.
2026-04-18T03:21:39.337699821Z INFO: Collecting host service logs for crio
2026-04-18T03:21:39.337699821Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-18T03:21:39.337946635Z INFO: Collecting host service logs for kubelet
2026-04-18T03:21:39.338241326Z INFO: Collecting host service logs for rpm-ostreed
2026-04-18T03:21:39.338673355Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-18T03:21:39.341709587Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-18T03:21:39.342001348Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-18T03:21:39.342336004Z INFO: Collecting host service logs for NetworkManager
2026-04-18T03:21:39.342633990Z INFO: Collecting host service logs for openvswitch
2026-04-18T03:21:39.342932637Z INFO: Collecting host service logs for ovs-configuration
2026-04-18T03:21:39.343237628Z INFO: Collecting host service logs for ovsdb-server
2026-04-18T03:21:39.344699380Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-18T03:21:39.346612606Z INFO: Waiting for worker host service log collection to complete ...
2026-04-18T03:21:39.356698260Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-18T03:21:39.361043343Z INFO: Gathering HAProxy config files
2026-04-18T03:21:39.361628234Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-18T03:21:39.390903255Z INFO: Waiting for node performance related collection to complete ...
2026-04-18T03:21:39.729506347Z Wrote inspect data to must-gather.
2026-04-18T03:21:39.729547033Z error: inspection completed with the errors occurred while gathering data:
2026-04-18T03:21:39.729547033Z     namespaces "assisted-installer" not found
2026-04-18T03:21:39.994550679Z error: the server doesn't have a resource type "performanceprofile"
2026-04-18T03:21:40.030743401Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-18T03:21:40.071106753Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-18T03:21:40.477085044Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-25mgk
2026-04-18T03:21:40.477137459Z insights-runtime-extractor-5hpwv
2026-04-18T03:21:40.477155475Z insights-runtime-extractor-dgzpp
2026-04-18T03:21:40.477173387Z insights-runtime-extractor-qct7q
2026-04-18T03:21:40.519037794Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-18T03:21:40.663366469Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-18T03:21:40.671950903Z INFO: "metallb-operator" not detected. Skipping.
2026-04-18T03:21:40.703986460Z No resources found in openshift-etcd namespace.
2026-04-18T03:21:40.743164594Z Wrote inspect data to must-gather.
2026-04-18T03:21:40.776593449Z No resources found
2026-04-18T03:21:40.830981757Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-18T03:21:40.989570692Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-18T03:21:41.071061198Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-18T03:21:41.078574114Z No resources found
2026-04-18T03:21:41.122334033Z INFO: Worker host service log collection to complete.
2026-04-18T03:21:41.180685929Z Wrote inspect data to must-gather.
2026-04-18T03:21:41.229176973Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-18T03:21:41.229217304Z INFO: on-disk MachineConfig config collection complete.
2026-04-18T03:21:41.382400654Z ERROR: No running kube-apiserver pods found
2026-04-18T03:21:41.389518353Z Wrote inspect data to must-gather.
2026-04-18T03:21:41.777748696Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-18T03:21:41.777748696Z See 'oc rsync -h' for help and examples
2026-04-18T03:21:41.820459965Z tar: Removing leading `/' from member names
2026-04-18T03:21:41.826897888Z INFO: HAProxy config collection complete.
2026-04-18T03:21:41.835151824Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-18T03:21:41.879826990Z error: the server doesn't have a resource type "machineconfigs"
2026-04-18T03:21:41.930079142Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-18T03:21:41.971430231Z Wrote inspect data to must-gather.
2026-04-18T03:21:42.045501838Z Wrote inspect data to must-gather.
2026-04-18T03:21:42.154982546Z INFO: 'previous-logs' folder not found on ip-10-0-128-79.ec2.internal, skipping...
2026-04-18T03:21:42.171449997Z Wrote inspect data to must-gather.
2026-04-18T03:21:42.271724466Z Wrote inspect data to must-gather.
2026-04-18T03:21:42.295000210Z INFO: Getting rules from prometheus-k8s-0
2026-04-18T03:21:42.394217718Z Gathering data for ns/kuadrant-system...
2026-04-18T03:21:42.457201343Z Wrote inspect data to must-gather.
2026-04-18T03:21:42.528337081Z INFO: 'previous-logs' folder not found on ip-10-0-129-229.ec2.internal, skipping...
2026-04-18T03:21:42.600575817Z Wrote inspect data to must-gather.
2026-04-18T03:21:42.805658603Z INFO: Getting status/config from prometheus-k8s-0
2026-04-18T03:21:42.818568883Z Wrote inspect data to must-gather.
2026-04-18T03:21:42.837071142Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-18T03:21:42.896786388Z INFO: 'previous-logs' folder not found on ip-10-0-140-103.ec2.internal, skipping...
2026-04-18T03:21:42.896824684Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-18T03:21:42.896853154Z INFO: Machine Config Daemon termination log collection complete.
2026-04-18T03:21:43.134688029Z Wrote inspect data to must-gather.
2026-04-18T03:21:43.158572960Z Wrote inspect data to must-gather.
2026-04-18T03:21:43.326473707Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-18T03:21:43.501607820Z Wrote inspect data to must-gather.
2026-04-18T03:21:43.549456180Z INFO: INTERCONNECT MODE
2026-04-18T03:21:43.549508096Z INFO: Gathering ovn-kubernetes DBs
2026-04-18T03:21:43.640996147Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-18T03:21:43.708829886Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-18T03:21:43.784345490Z INFO: Gathering OVN_Northbound from ovnkube-node-cf2h2...
2026-04-18T03:21:43.784679634Z INFO: Gathering OVN_Northbound from ovnkube-node-k8fsv...
2026-04-18T03:21:43.784952354Z INFO: Gathering OVN_Northbound from ovnkube-node-lsz2j...
2026-04-18T03:21:43.785258955Z INFO: Gathering OVN_Southbound from ovnkube-node-cf2h2...
2026-04-18T03:21:43.785599659Z INFO: Gathering OVN_Southbound from ovnkube-node-k8fsv...
2026-04-18T03:21:43.785981240Z INFO: Gathering OVN_Southbound from ovnkube-node-lsz2j...
2026-04-18T03:21:43.843630943Z Wrote inspect data to must-gather.
2026-04-18T03:21:43.926269241Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-18T03:21:44.170442662Z tar: Removing leading `/' from member names
2026-04-18T03:21:44.170442662Z tar: Removing leading `/' from member names
2026-04-18T03:21:44.181064327Z tar: Removing leading `/' from member names
2026-04-18T03:21:44.247820469Z tar: Removing leading `/' from member names
2026-04-18T03:21:44.276496761Z tar: Removing leading `/' from member names
2026-04-18T03:21:44.324562550Z tar: Removing leading `/' from member names
2026-04-18T03:21:44.368803975Z W0418 03:21:44.367960    1261 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-18T03:21:44.389091759Z Wrote inspect data to must-gather.
2026-04-18T03:21:44.442563581Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-18T03:21:44.480656133Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-18T03:21:44.766832518Z Gathering data for ns/openshift-monitoring...
2026-04-18T03:21:44.855736798Z Wrote inspect data to must-gather.
2026-04-18T03:21:45.006286919Z INFO: Getting status from alertmanager-main-0
2026-04-18T03:21:45.030710047Z INFO: Waiting for network log collection to complete ...
2026-04-18T03:21:45.030743537Z INFO: Waiting for ovnk database copies to complete ...
2026-04-18T03:21:45.031218355Z INFO: Copying ovnk databases complete.
2026-04-18T03:21:45.034424356Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-04-18T03:21:45.036691168Z ovnk_database_store/
2026-04-18T03:21:45.036691168Z ovnk_database_store/ovnkube-node-cf2h2_nbdb
2026-04-18T03:21:45.053186886Z ovnk_database_store/ovnkube-node-cf2h2_sbdb
2026-04-18T03:21:45.085796672Z ovnk_database_store/ovnkube-node-lsz2j_sbdb
2026-04-18T03:21:45.108032990Z ovnk_database_store/ovnkube-node-k8fsv_nbdb
2026-04-18T03:21:45.120205176Z ovnk_database_store/ovnkube-node-lsz2j_nbdb
2026-04-18T03:21:45.131974773Z ovnk_database_store/ovnkube-node-k8fsv_sbdb
2026-04-18T03:21:45.165237621Z INFO: Network log collection complete.
2026-04-18T03:21:45.218351370Z Wrote inspect data to must-gather.
2026-04-18T03:21:45.516528585Z Wrote inspect data to must-gather.
2026-04-18T03:21:45.797402257Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-18T03:21:45.801403414Z INFO: Fallback to identify the container image from release info
2026-04-18T03:21:46.365911158Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-18T03:21:46.502623022Z daemonset.apps/perf-node-gather-daemonset created
2026-04-18T03:21:46.661210987Z Waiting for performance profile collector pods to become ready: 1
2026-04-18T03:21:46.939851748Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-18T03:21:46.953174010Z Gathering data for ns/openshift-network-console...
2026-04-18T03:21:47.458591279Z Gathering data for ns/openshift-console-operator...
2026-04-18T03:21:47.803101505Z Waiting for performance profile collector pods to become ready: 2
2026-04-18T03:21:47.896873382Z Gathering data for ns/openshift-console...
2026-04-18T03:21:48.401774466Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-18T03:21:48.571333556Z Wrote inspect data to must-gather.
2026-04-18T03:21:48.779555618Z Gathering data for ns/openshift-dns-operator...
2026-04-18T03:21:48.935110099Z Waiting for performance profile collector pods to become ready: 3
2026-04-18T03:21:49.099480858Z Gathering data for ns/openshift-dns...
2026-04-18T03:21:49.676273819Z Gathering data for ns/openshift-image-registry...
2026-04-18T03:21:50.057363765Z Waiting for performance profile collector pods to become ready: 4
2026-04-18T03:21:50.159130021Z Gathering data for ns/openshift-ingress-operator...
2026-04-18T03:21:50.505656499Z Gathering data for ns/openshift-ingress...
2026-04-18T03:21:51.198561640Z Waiting for performance profile collector pods to become ready: 5
2026-04-18T03:21:51.220771663Z Gathering data for ns/openshift-ingress-canary...
2026-04-18T03:21:51.657429792Z Gathering data for ns/openshift-insights...
2026-04-18T03:21:51.949704864Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-18T03:21:52.321492751Z Waiting for performance profile collector pods to become ready: 6
2026-04-18T03:21:53.442941355Z Waiting for performance profile collector pods to become ready: 7
2026-04-18T03:21:53.585447488Z Gathering data for ns/opendatahub...
2026-04-18T03:21:54.557637164Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-18T03:21:54.804282323Z Collecting performance related data for node ip-10-0-128-79.ec2.internal
2026-04-18T03:21:54.810983904Z Collecting performance related data for node ip-10-0-129-229.ec2.internal
2026-04-18T03:21:54.818328035Z Collecting performance related data for node ip-10-0-140-103.ec2.internal
2026-04-18T03:21:55.082685892Z Gathering data for ns/openshift-lws-operator...
2026-04-18T03:21:56.476453491Z Gathering data for ns/openshift-config...
2026-04-18T03:21:56.778129844Z Gathering data for ns/openshift-config-managed...
2026-04-18T03:21:56.957258347Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-18T03:21:57.288054189Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-18T03:21:57.619542190Z Gathering data for ns/openshift-kube-apiserver...
2026-04-18T03:21:57.939575554Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-18T03:21:58.283522088Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-18T03:21:58.602541707Z Gathering data for ns/openshift-kube-scheduler...
2026-04-18T03:21:58.899570513Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-18T03:21:59.233727151Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-18T03:21:59.596909145Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-18T03:21:59.935705356Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-18T03:22:00.585206231Z Gathering data for ns/openshift-multus...
2026-04-18T03:22:01.955667603Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-18T03:22:01.965114624Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-18T03:22:03.558323552Z Collecting kubelet logs for node ip-10-0-128-79.ec2.internal
2026-04-18T03:22:03.561022036Z Collecting kubelet logs for node ip-10-0-129-229.ec2.internal
2026-04-18T03:22:03.563107858Z Gathering data for ns/openshift-host-network...
2026-04-18T03:22:03.563107858Z Collecting kubelet logs for node ip-10-0-140-103.ec2.internal
2026-04-18T03:22:03.948429049Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-18T03:22:03.952896678Z INFO: Node performance data collection complete.
2026-04-18T03:22:03.988496228Z Gathering data for ns/openshift-network-diagnostics...
2026-04-18T03:22:04.555608034Z Gathering data for ns/openshift-network-node-identity...
2026-04-18T03:22:05.021230483Z Gathering data for ns/openshift-network-operator...
2026-04-18T03:22:05.394031670Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-18T03:22:05.677821165Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-18T03:22:06.142689386Z Gathering data for ns/openshift-apiserver-operator...
2026-04-18T03:22:06.454605435Z Gathering data for ns/openshift-apiserver...
2026-04-18T03:22:06.872601557Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-18T03:22:06.971759223Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-04-18T03:22:07.176540611Z Gathering data for ns/openshift-controller-manager...
2026-04-18T03:22:07.534899255Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-18T03:22:08.195772509Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-18T03:22:08.513147525Z Gathering data for ns/openshift-service-ca-operator...
2026-04-18T03:22:08.850211802Z Gathering data for ns/openshift-service-ca...
2026-04-18T03:22:09.241934500Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-18T03:22:10.318054303Z Wrote inspect data to must-gather.
2026-04-18T03:22:10.318102645Z error: inspection completed with the errors occurred while gathering data:
2026-04-18T03:22:10.318102645Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-18T03:22:10.460998887Z error: the server doesn't have a resource type "clusters"
2026-04-18T03:22:10.643839780Z Caches written to disk