2026-05-28T18:19:19.103665839Z [disk usage checker] Started
2026-05-28T18:19:19.107763249Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T18:19:19.340720330Z Gathering data for ns/openshift-cluster-version...
2026-05-28T18:19:19.513038578Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T18:19:19.609851538Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T18:19:19.719922523Z Gathering data for ns/default...
2026-05-28T18:19:20.100864762Z Gathering data for ns/openshift...
2026-05-28T18:19:20.570239970Z Gathering data for ns/kube-system...
2026-05-28T18:19:21.321317157Z Gathering data for ns/openshift-etcd...
2026-05-28T18:19:21.651617392Z Waiting on subprocesses to finish execution.
2026-05-28T18:19:21.662972198Z INFO: Gathering HAProxy config files
2026-05-28T18:19:21.664877870Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-28T18:19:21.669265139Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-28T18:19:21.672899543Z INFO: Collecting host service logs for crio
2026-05-28T18:19:21.673330018Z INFO: Collecting host service logs for kubelet
2026-05-28T18:19:21.673854709Z INFO: Collecting host service logs for rpm-ostreed
2026-05-28T18:19:21.674206717Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-28T18:19:21.674548233Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-28T18:19:21.674883360Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-28T18:19:21.675224713Z INFO: Collecting host service logs for NetworkManager
2026-05-28T18:19:21.675717734Z INFO: Collecting host service logs for openvswitch
2026-05-28T18:19:21.676170609Z INFO: Collecting host service logs for ovs-configuration
2026-05-28T18:19:21.676450649Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-28T18:19:21.676571393Z INFO: Collecting host service logs for ovsdb-server
2026-05-28T18:19:21.676922818Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-28T18:19:21.677538754Z INFO: Waiting for worker host service log collection to complete ...
2026-05-28T18:19:21.701946599Z Wrote inspect data to must-gather.
2026-05-28T18:19:21.702042467Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T18:19:21.702042467Z     namespaces "assisted-installer" not found
2026-05-28T18:19:21.791076765Z INFO: Waiting for node performance related collection to complete ...
2026-05-28T18:19:22.334476545Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-28T18:19:22.402773012Z No resources found
2026-05-28T18:19:22.475395894Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-28T18:19:22.693080786Z error: the server doesn't have a resource type "performanceprofile"
2026-05-28T18:19:22.739257556Z Wrote inspect data to must-gather.
2026-05-28T18:19:22.805778769Z No resources found in openshift-etcd namespace.
2026-05-28T18:19:22.885434441Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-28T18:19:22.890070628Z INFO: "metallb-operator" not detected. Skipping.
2026-05-28T18:19:22.900497338Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-7dfkj
2026-05-28T18:19:22.900540441Z insights-runtime-extractor-6gmkr
2026-05-28T18:19:22.900560541Z insights-runtime-extractor-9fg6h
2026-05-28T18:19:22.900579818Z insights-runtime-extractor-vgg6n
2026-05-28T18:19:22.990592772Z INFO: Worker host service log collection to complete.
2026-05-28T18:19:23.093220158Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-28T18:19:23.096252432Z INFO: on-disk MachineConfig config collection complete.
2026-05-28T18:19:23.200134540Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-28T18:19:23.221008878Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-28T18:19:23.221008878Z See 'oc rsync -h' for help and examples
2026-05-28T18:19:23.253220544Z ERROR: No running kube-apiserver pods found
2026-05-28T18:19:23.375640126Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-28T18:19:23.560514087Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-28T18:19:23.757804192Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-28T18:19:23.957387755Z Wrote inspect data to must-gather.
2026-05-28T18:19:23.960464786Z INFO: 'previous-logs' folder not found on ip-10-0-132-117.ec2.internal, skipping...
2026-05-28T18:19:24.186495833Z No resources found
2026-05-28T18:19:24.188203593Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T18:19:24.240360496Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T18:19:24.286966135Z Wrote inspect data to must-gather.
2026-05-28T18:19:24.482018553Z Wrote inspect data to must-gather.
2026-05-28T18:19:24.489587247Z Gathering data for ns/kuadrant-system...
2026-05-28T18:19:24.530851887Z Wrote inspect data to must-gather.
2026-05-28T18:19:24.537128558Z Wrote inspect data to must-gather.
2026-05-28T18:19:24.609328564Z error: the server doesn't have a resource type "machineconfigs"
2026-05-28T18:19:24.648626152Z INFO: 'previous-logs' folder not found on ip-10-0-135-41.ec2.internal, skipping...
2026-05-28T18:19:24.649583688Z tar: Removing leading `/' from member names
2026-05-28T18:19:24.655545855Z INFO: HAProxy config collection complete.
2026-05-28T18:19:24.686714141Z INFO: Getting rules from prometheus-k8s-0
2026-05-28T18:19:24.961195771Z Wrote inspect data to must-gather.
2026-05-28T18:19:24.962983332Z Wrote inspect data to must-gather.
2026-05-28T18:19:24.982628572Z INFO: 'previous-logs' folder not found on ip-10-0-137-72.ec2.internal, skipping...
2026-05-28T18:19:24.982766257Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-28T18:19:24.982849951Z INFO: Machine Config Daemon termination log collection complete.
2026-05-28T18:19:25.205959967Z INFO: Getting status/config from prometheus-k8s-0
2026-05-28T18:19:25.268488902Z Wrote inspect data to must-gather.
2026-05-28T18:19:25.317181241Z Wrote inspect data to must-gather.
2026-05-28T18:19:25.475660852Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-28T18:19:25.680544330Z Wrote inspect data to must-gather.
2026-05-28T18:19:25.709461717Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-28T18:19:25.812725552Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-28T18:19:25.832404852Z Wrote inspect data to must-gather.
2026-05-28T18:19:25.994374816Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-28T18:19:26.104811356Z INFO: INTERCONNECT MODE
2026-05-28T18:19:26.104854197Z INFO: Gathering ovn-kubernetes DBs
2026-05-28T18:19:26.133334866Z Wrote inspect data to must-gather.
2026-05-28T18:19:26.267655441Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-28T18:19:26.332311871Z INFO: Gathering OVN_Northbound from ovnkube-node-fsvt7...
2026-05-28T18:19:26.332600003Z INFO: Gathering OVN_Northbound from ovnkube-node-sjcbp...
2026-05-28T18:19:26.332822298Z INFO: Gathering OVN_Northbound from ovnkube-node-sphfp...
2026-05-28T18:19:26.333161421Z INFO: Gathering OVN_Southbound from ovnkube-node-fsvt7...
2026-05-28T18:19:26.333442541Z INFO: Gathering OVN_Southbound from ovnkube-node-sjcbp...
2026-05-28T18:19:26.333703511Z INFO: Gathering OVN_Southbound from ovnkube-node-sphfp...
2026-05-28T18:19:26.498155614Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-28T18:19:26.633092864Z Wrote inspect data to must-gather.
2026-05-28T18:19:26.688702806Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-28T18:19:26.770962518Z tar: Removing leading `/' from member names
2026-05-28T18:19:26.807605620Z tar: Removing leading `/' from member names
2026-05-28T18:19:26.810640905Z tar: Removing leading `/' from member names
2026-05-28T18:19:26.867263931Z tar: Removing leading `/' from member names
2026-05-28T18:19:26.877552781Z tar: Removing leading `/' from member names
2026-05-28T18:19:26.878866973Z tar: Removing leading `/' from member names
2026-05-28T18:19:26.939691411Z Gathering data for ns/openshift-monitoring...
2026-05-28T18:19:27.096638204Z Wrote inspect data to must-gather.
2026-05-28T18:19:27.302573172Z INFO: Getting status from alertmanager-main-0
2026-05-28T18:19:27.503844433Z Wrote inspect data to must-gather.
2026-05-28T18:19:27.576219588Z INFO: Waiting for network log collection to complete ...
2026-05-28T18:19:27.576289391Z INFO: Waiting for ovnk database copies to complete ...
2026-05-28T18:19:27.576677019Z INFO: Copying ovnk databases complete.
2026-05-28T18:19:27.578663129Z 4.2M	must-gather/network_logs/ovnk_database_store
2026-05-28T18:19:27.581734209Z ovnk_database_store/
2026-05-28T18:19:27.581734209Z ovnk_database_store/ovnkube-node-fsvt7_sbdb
2026-05-28T18:19:27.631109696Z ovnk_database_store/ovnkube-node-fsvt7_nbdb
2026-05-28T18:19:27.643172309Z ovnk_database_store/ovnkube-node-sphfp_nbdb
2026-05-28T18:19:27.657413374Z ovnk_database_store/ovnkube-node-sphfp_sbdb
2026-05-28T18:19:27.683270511Z ovnk_database_store/ovnkube-node-sjcbp_sbdb
2026-05-28T18:19:27.703733463Z ovnk_database_store/ovnkube-node-sjcbp_nbdb
2026-05-28T18:19:27.721084018Z INFO: Network log collection complete.
2026-05-28T18:19:27.841426531Z Wrote inspect data to must-gather.
2026-05-28T18:19:28.122658078Z Wrote inspect data to must-gather.
2026-05-28T18:19:28.426206135Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-28T18:19:28.428527329Z INFO: Fallback to identify the container image from release info
2026-05-28T18:19:29.021145766Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-28T18:19:29.091835808Z Gathering data for ns/openshift-network-console...
2026-05-28T18:19:29.176653810Z daemonset.apps/perf-node-gather-daemonset created
2026-05-28T18:19:29.201027750Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T18:19:29.341758793Z Waiting for performance profile collector pods to become ready: 1
2026-05-28T18:19:29.635264296Z Gathering data for ns/openshift-console-operator...
2026-05-28T18:19:30.111280525Z Gathering data for ns/openshift-console...
2026-05-28T18:19:30.500277708Z Waiting for performance profile collector pods to become ready: 2
2026-05-28T18:19:30.620833905Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-28T18:19:31.040336497Z Gathering data for ns/openshift-dns-operator...
2026-05-28T18:19:31.429953947Z Gathering data for ns/openshift-dns...
2026-05-28T18:19:31.602741700Z Wrote inspect data to must-gather.
2026-05-28T18:19:31.622863107Z Waiting for performance profile collector pods to become ready: 3
2026-05-28T18:19:32.050021525Z Gathering data for ns/openshift-image-registry...
2026-05-28T18:19:32.518790675Z Gathering data for ns/openshift-ingress-operator...
2026-05-28T18:19:32.754754900Z Waiting for performance profile collector pods to become ready: 4
2026-05-28T18:19:32.913690757Z Gathering data for ns/openshift-ingress...
2026-05-28T18:19:33.684187250Z Gathering data for ns/openshift-ingress-canary...
2026-05-28T18:19:33.893049804Z Waiting for performance profile collector pods to become ready: 5
2026-05-28T18:19:34.160978515Z Gathering data for ns/openshift-insights...
2026-05-28T18:19:34.206719273Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T18:19:35.010613577Z Waiting for performance profile collector pods to become ready: 6
2026-05-28T18:19:36.121504076Z Waiting for performance profile collector pods to become ready: 7
2026-05-28T18:19:36.130568385Z Gathering data for ns/opendatahub...
2026-05-28T18:19:37.243875863Z Waiting for performance profile collector pods to become ready: 8
2026-05-28T18:19:37.616162690Z Gathering data for ns/openshift-lws-operator...
2026-05-28T18:19:38.362582930Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-28T18:19:38.635556238Z Collecting performance related data for node ip-10-0-137-72.ec2.internal
2026-05-28T18:19:38.643360187Z Collecting performance related data for node ip-10-0-132-117.ec2.internal
2026-05-28T18:19:38.651659827Z Collecting performance related data for node ip-10-0-135-41.ec2.internal
2026-05-28T18:19:38.896671855Z Gathering data for ns/openshift-config...
2026-05-28T18:19:39.212283832Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T18:19:39.280464282Z Gathering data for ns/openshift-config-managed...
2026-05-28T18:19:39.866058369Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-28T18:19:40.289547064Z Gathering data for ns/openshift-kube-apiserver...
2026-05-28T18:19:40.604321714Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-28T18:19:40.920062031Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-28T18:19:41.305653011Z Gathering data for ns/openshift-kube-scheduler...
2026-05-28T18:19:41.622018944Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-28T18:19:42.044095726Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-28T18:19:42.416742580Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-28T18:19:42.781655804Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-28T18:19:43.449062408Z Gathering data for ns/openshift-multus...
2026-05-28T18:19:44.219625883Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T18:19:44.963545278Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-28T18:19:46.536797457Z Gathering data for ns/openshift-host-network...
2026-05-28T18:19:46.906189748Z Gathering data for ns/openshift-network-diagnostics...
2026-05-28T18:19:47.315680281Z Collecting kubelet logs for node ip-10-0-137-72.ec2.internal
2026-05-28T18:19:47.317894321Z Collecting kubelet logs for node ip-10-0-132-117.ec2.internal
2026-05-28T18:19:47.320407643Z Collecting kubelet logs for node ip-10-0-135-41.ec2.internal
2026-05-28T18:19:47.561059802Z Gathering data for ns/openshift-network-node-identity...
2026-05-28T18:19:47.725190540Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-28T18:19:47.730596652Z INFO: Node performance data collection complete.
2026-05-28T18:19:48.072190918Z Gathering data for ns/openshift-network-operator...
2026-05-28T18:19:48.489685694Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-28T18:19:48.779639369Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-28T18:19:49.226261389Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-05-28T18:19:49.306338164Z Gathering data for ns/openshift-apiserver-operator...
2026-05-28T18:19:49.673718842Z Gathering data for ns/openshift-apiserver...
2026-05-28T18:19:50.129202997Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-28T18:19:50.434661123Z Gathering data for ns/openshift-controller-manager...
2026-05-28T18:19:50.755042455Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-28T18:19:51.440601393Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-28T18:19:51.743399245Z Gathering data for ns/openshift-service-ca-operator...
2026-05-28T18:19:52.075334932Z Gathering data for ns/openshift-service-ca...
2026-05-28T18:19:52.468014692Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-28T18:19:53.520006023Z Wrote inspect data to must-gather.
2026-05-28T18:19:53.520049933Z error: inspection completed with the errors occurred while gathering data:
2026-05-28T18:19:53.520049933Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-28T18:19:53.657424568Z error: the server doesn't have a resource type "clusters"
2026-05-28T18:19:53.840208632Z Caches written to disk