2026-04-24T17:29:58.647687931Z [disk usage checker] Started
2026-04-24T17:29:58.651108894Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-24T17:29:58.876849014Z Gathering data for ns/openshift-cluster-version...
2026-04-24T17:29:59.032616808Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T17:29:59.133313510Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T17:29:59.235815119Z Gathering data for ns/default...
2026-04-24T17:29:59.592007890Z Gathering data for ns/openshift...
2026-04-24T17:30:00.049480858Z Gathering data for ns/kube-system...
2026-04-24T17:30:00.726531442Z Gathering data for ns/openshift-etcd...
2026-04-24T17:30:01.086425106Z Wrote inspect data to must-gather.
2026-04-24T17:30:01.086512034Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T17:30:01.086512034Z     namespaces "assisted-installer" not found
2026-04-24T17:30:01.248476860Z Waiting on subprocesses to finish execution.
2026-04-24T17:30:01.257000961Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-24T17:30:01.265117696Z INFO: Gathering HAProxy config files
2026-04-24T17:30:01.267041543Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-24T17:30:01.273495189Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-24T17:30:01.314525160Z INFO: Collecting host service logs for crio
2026-04-24T17:30:01.314902837Z INFO: Collecting host service logs for kubelet
2026-04-24T17:30:01.315259464Z INFO: Collecting host service logs for rpm-ostreed
2026-04-24T17:30:01.315599599Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-24T17:30:01.315983614Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-24T17:30:01.316315349Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-24T17:30:01.316646039Z INFO: Collecting host service logs for NetworkManager
2026-04-24T17:30:01.316997612Z INFO: Collecting host service logs for openvswitch
2026-04-24T17:30:01.317319625Z INFO: Collecting host service logs for ovs-configuration
2026-04-24T17:30:01.317635991Z INFO: Collecting host service logs for ovsdb-server
2026-04-24T17:30:01.317979966Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-24T17:30:01.318373030Z INFO: Waiting for worker host service log collection to complete ...
2026-04-24T17:30:01.356995569Z INFO: Waiting for node performance related collection to complete ...
2026-04-24T17:30:01.804243271Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-24T17:30:02.035848688Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-24T17:30:02.222904486Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-24T17:30:02.223028332Z INFO: on-disk MachineConfig config collection complete.
2026-04-24T17:30:02.237458397Z No resources found in openshift-etcd namespace.
2026-04-24T17:30:02.443956652Z INFO: "metallb-operator" not detected. Skipping.
2026-04-24T17:30:02.601083035Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-24T17:30:02.603101949Z No resources found
2026-04-24T17:30:02.650637324Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-4wsnt
2026-04-24T17:30:02.650637324Z insights-runtime-extractor-d7n6q
2026-04-24T17:30:02.650637324Z insights-runtime-extractor-fkrh6
2026-04-24T17:30:02.650637324Z insights-runtime-extractor-xlfzt
2026-04-24T17:30:02.663055371Z error: the server doesn't have a resource type "performanceprofile"
2026-04-24T17:30:02.708916892Z Wrote inspect data to must-gather.
2026-04-24T17:30:02.732700127Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-24T17:30:02.781618186Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-24T17:30:03.029460337Z INFO: Worker host service log collection to complete.
2026-04-24T17:30:03.109638152Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-24T17:30:03.227584983Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-24T17:30:03.227584983Z See 'oc rsync -h' for help and examples
2026-04-24T17:30:03.531471324Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T17:30:03.666256021Z No resources found
2026-04-24T17:30:03.879089931Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-24T17:30:03.905945113Z Wrote inspect data to must-gather.
2026-04-24T17:30:03.934882465Z ERROR: No running kube-apiserver pods found
2026-04-24T17:30:03.950025124Z Wrote inspect data to must-gather.
2026-04-24T17:30:04.046160486Z INFO: 'previous-logs' folder not found on ip-10-0-129-227.ec2.internal, skipping...
2026-04-24T17:30:04.156379624Z Wrote inspect data to must-gather.
2026-04-24T17:30:04.193427683Z Wrote inspect data to must-gather.
2026-04-24T17:30:04.273539318Z tar: Removing leading `/' from member names
2026-04-24T17:30:04.280235173Z INFO: HAProxy config collection complete.
2026-04-24T17:30:04.304487157Z Wrote inspect data to must-gather.
2026-04-24T17:30:04.380028541Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-24T17:30:04.401874578Z INFO: Getting rules from prometheus-k8s-0
2026-04-24T17:30:04.558503759Z Gathering data for ns/kuadrant-system...
2026-04-24T17:30:04.622768474Z error: the server doesn't have a resource type "machineconfigs"
2026-04-24T17:30:04.633330525Z INFO: 'previous-logs' folder not found on ip-10-0-137-69.ec2.internal, skipping...
2026-04-24T17:30:04.642944592Z Wrote inspect data to must-gather.
2026-04-24T17:30:04.966443622Z Wrote inspect data to must-gather.
2026-04-24T17:30:04.994775997Z INFO: Getting status/config from prometheus-k8s-0
2026-04-24T17:30:05.042664282Z INFO: 'previous-logs' folder not found on ip-10-0-139-51.ec2.internal, skipping...
2026-04-24T17:30:05.042821870Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-24T17:30:05.042886922Z INFO: Machine Config Daemon termination log collection complete.
2026-04-24T17:30:05.098197948Z Wrote inspect data to must-gather.
2026-04-24T17:30:05.193127497Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-24T17:30:05.308328324Z Wrote inspect data to must-gather.
2026-04-24T17:30:05.465374156Z Wrote inspect data to must-gather.
2026-04-24T17:30:05.537610742Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-24T17:30:05.565797501Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-24T17:30:05.569682283Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-24T17:30:05.778831009Z Gathering data for ns/openshift-monitoring...
2026-04-24T17:30:05.884303254Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-24T17:30:05.909129369Z INFO: INTERCONNECT MODE
2026-04-24T17:30:05.909178253Z INFO: Gathering ovn-kubernetes DBs
2026-04-24T17:30:05.926896484Z Wrote inspect data to must-gather.
2026-04-24T17:30:06.116350896Z INFO: Gathering OVN_Northbound from ovnkube-node-7fpht...
2026-04-24T17:30:06.116561584Z INFO: Gathering OVN_Northbound from ovnkube-node-dmn7d...
2026-04-24T17:30:06.116851162Z INFO: Gathering OVN_Northbound from ovnkube-node-xxfbv...
2026-04-24T17:30:06.117205832Z INFO: Gathering OVN_Southbound from ovnkube-node-7fpht...
2026-04-24T17:30:06.117507844Z INFO: Gathering OVN_Southbound from ovnkube-node-dmn7d...
2026-04-24T17:30:06.117793097Z INFO: Gathering OVN_Southbound from ovnkube-node-xxfbv...
2026-04-24T17:30:06.238611732Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-24T17:30:06.363061040Z Wrote inspect data to must-gather.
2026-04-24T17:30:06.476403860Z tar: Removing leading `/' from member names
2026-04-24T17:30:06.516706286Z tar: Removing leading `/' from member names
2026-04-24T17:30:06.546413014Z tar: Removing leading `/' from member names
2026-04-24T17:30:06.551049078Z tar: Removing leading `/' from member names
2026-04-24T17:30:06.565894473Z tar: Removing leading `/' from member names
2026-04-24T17:30:06.609762675Z tar: Removing leading `/' from member names
2026-04-24T17:30:06.629473308Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-24T17:30:06.789078824Z Wrote inspect data to must-gather.
2026-04-24T17:30:07.119975495Z Wrote inspect data to must-gather.
2026-04-24T17:30:07.122231175Z INFO: Getting status from alertmanager-main-0
2026-04-24T17:30:07.149044634Z INFO: Waiting for network log collection to complete ...
2026-04-24T17:30:07.149166589Z INFO: Waiting for ovnk database copies to complete ...
2026-04-24T17:30:07.149581381Z INFO: Copying ovnk databases complete.
2026-04-24T17:30:07.151714392Z 5.0M	must-gather/network_logs/ovnk_database_store
2026-04-24T17:30:07.162635396Z ovnk_database_store/
2026-04-24T17:30:07.162755306Z ovnk_database_store/ovnkube-node-xxfbv_nbdb
2026-04-24T17:30:07.179079111Z ovnk_database_store/ovnkube-node-dmn7d_nbdb
2026-04-24T17:30:07.193593901Z ovnk_database_store/ovnkube-node-7fpht_nbdb
2026-04-24T17:30:07.207971833Z ovnk_database_store/ovnkube-node-dmn7d_sbdb
2026-04-24T17:30:07.238949496Z ovnk_database_store/ovnkube-node-7fpht_sbdb
2026-04-24T17:30:07.286627685Z ovnk_database_store/ovnkube-node-xxfbv_sbdb
2026-04-24T17:30:07.318287882Z INFO: Network log collection complete.
2026-04-24T17:30:07.413654136Z Wrote inspect data to must-gather.
2026-04-24T17:30:07.684271847Z Wrote inspect data to must-gather.
2026-04-24T17:30:08.035854236Z Wrote inspect data to must-gather.
2026-04-24T17:30:08.113404005Z Gathering data for ns/openshift-network-console...
2026-04-24T17:30:08.367645080Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-24T17:30:08.370654910Z INFO: Fallback to identify the container image from release info
2026-04-24T17:30:08.635253135Z Gathering data for ns/openshift-console-operator...
2026-04-24T17:30:08.900137975Z Wrote inspect data to must-gather.
2026-04-24T17:30:08.914060473Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-24T17:30:09.049364881Z Gathering data for ns/openshift-console...
2026-04-24T17:30:09.212098233Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-24T17:30:09.336513079Z daemonset.apps/perf-node-gather-daemonset created
2026-04-24T17:30:09.491422301Z Waiting for performance profile collector pods to become ready: 1
2026-04-24T17:30:09.554503457Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-24T17:30:09.960599822Z Gathering data for ns/openshift-dns-operator...
2026-04-24T17:30:10.294544599Z Gathering data for ns/openshift-dns...
2026-04-24T17:30:10.636091494Z Waiting for performance profile collector pods to become ready: 2
2026-04-24T17:30:10.942838467Z Gathering data for ns/openshift-image-registry...
2026-04-24T17:30:11.421269224Z Gathering data for ns/openshift-ingress-operator...
2026-04-24T17:30:11.785214501Z Waiting for performance profile collector pods to become ready: 3
2026-04-24T17:30:11.791178406Z Gathering data for ns/openshift-ingress...
2026-04-24T17:30:12.307616999Z Gathering data for ns/openshift-ingress-canary...
2026-04-24T17:30:12.758904430Z Gathering data for ns/openshift-insights...
2026-04-24T17:30:12.902954213Z Waiting for performance profile collector pods to become ready: 4
2026-04-24T17:30:13.919179729Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-24T17:30:14.022724516Z Waiting for performance profile collector pods to become ready: 5
2026-04-24T17:30:15.145551584Z Waiting for performance profile collector pods to become ready: 6
2026-04-24T17:30:15.549281140Z Gathering data for ns/openshift-lws-operator...
2026-04-24T17:30:16.096755216Z Gathering data for ns/kserve...
2026-04-24T17:30:16.301756651Z Waiting for performance profile collector pods to become ready: 7
2026-04-24T17:30:17.429068518Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-24T17:30:17.693348528Z Collecting performance related data for node ip-10-0-137-69.ec2.internal
2026-04-24T17:30:17.700548111Z Collecting performance related data for node ip-10-0-139-51.ec2.internal
2026-04-24T17:30:17.709257602Z Collecting performance related data for node ip-10-0-129-227.ec2.internal
2026-04-24T17:30:17.826213018Z Gathering data for ns/openshift-config...
2026-04-24T17:30:18.208547478Z Gathering data for ns/openshift-config-managed...
2026-04-24T17:30:18.839215626Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-24T17:30:18.927443651Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-24T17:30:19.228019899Z Gathering data for ns/openshift-kube-apiserver...
2026-04-24T17:30:19.595992935Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-24T17:30:19.942891667Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-24T17:30:20.319301770Z Gathering data for ns/openshift-kube-scheduler...
2026-04-24T17:30:20.681707759Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-24T17:30:21.087720190Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-24T17:30:21.485625478Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-24T17:30:21.847030894Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-24T17:30:22.525588553Z Gathering data for ns/openshift-multus...
2026-04-24T17:30:23.933524550Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-24T17:30:24.000997982Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-24T17:30:25.578255836Z Gathering data for ns/openshift-host-network...
2026-04-24T17:30:25.933288331Z Gathering data for ns/openshift-network-diagnostics...
2026-04-24T17:30:26.478987071Z Collecting kubelet logs for node ip-10-0-137-69.ec2.internal
2026-04-24T17:30:26.481092022Z Collecting kubelet logs for node ip-10-0-139-51.ec2.internal
2026-04-24T17:30:26.483037592Z Collecting kubelet logs for node ip-10-0-129-227.ec2.internal
2026-04-24T17:30:26.525849696Z Gathering data for ns/openshift-network-node-identity...
2026-04-24T17:30:26.918547631Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-24T17:30:26.924505696Z INFO: Node performance data collection complete.
2026-04-24T17:30:27.094831999Z Gathering data for ns/openshift-network-operator...
2026-04-24T17:30:27.542317300Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-24T17:30:27.865940289Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-24T17:30:28.398274782Z Gathering data for ns/openshift-apiserver-operator...
2026-04-24T17:30:28.756660503Z Gathering data for ns/openshift-apiserver...
2026-04-24T17:30:28.940029840Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-04-24T17:30:29.195627843Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-24T17:30:29.521634166Z Gathering data for ns/openshift-controller-manager...
2026-04-24T17:30:29.829118032Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-24T17:30:30.499801669Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-24T17:30:30.845294869Z Gathering data for ns/openshift-service-ca-operator...
2026-04-24T17:30:31.189784905Z Gathering data for ns/openshift-service-ca...
2026-04-24T17:30:31.610127595Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-24T17:30:32.607938173Z Wrote inspect data to must-gather.
2026-04-24T17:30:32.607988803Z error: inspection completed with the errors occurred while gathering data:
2026-04-24T17:30:32.607988803Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-24T17:30:32.751454892Z error: the server doesn't have a resource type "clusters"
2026-04-24T17:30:32.925307104Z Caches written to disk