2026-04-23T09:27:55.617258576Z [disk usage checker] Started
2026-04-23T09:27:55.663204434Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:27:55.959674515Z Gathering data for ns/openshift-cluster-version...
2026-04-23T09:27:56.108939489Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T09:27:56.202950304Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T09:27:56.293204305Z Gathering data for ns/default...
2026-04-23T09:27:56.602182477Z Gathering data for ns/openshift...
2026-04-23T09:27:57.034850082Z Gathering data for ns/kube-system...
2026-04-23T09:27:57.790953480Z Gathering data for ns/openshift-etcd...
2026-04-23T09:27:58.139199441Z Wrote inspect data to must-gather.
2026-04-23T09:27:58.139235457Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T09:27:58.139235457Z     namespaces "assisted-installer" not found
2026-04-23T09:27:58.175944309Z Waiting on subprocesses to finish execution.
2026-04-23T09:27:58.187668695Z INFO: Gathering HAProxy config files
2026-04-23T09:27:58.202493128Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-23T09:27:58.207060403Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-23T09:27:58.219515683Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-23T09:27:58.251167076Z INFO: Collecting host service logs for crio
2026-04-23T09:27:58.251580455Z INFO: Collecting host service logs for kubelet
2026-04-23T09:27:58.251980791Z INFO: Collecting host service logs for rpm-ostreed
2026-04-23T09:27:58.252351044Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-23T09:27:58.252725197Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-23T09:27:58.253086765Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-23T09:27:58.253462530Z INFO: Collecting host service logs for NetworkManager
2026-04-23T09:27:58.253817049Z INFO: Collecting host service logs for openvswitch
2026-04-23T09:27:58.254152661Z INFO: Collecting host service logs for ovs-configuration
2026-04-23T09:27:58.254531844Z INFO: Collecting host service logs for ovsdb-server
2026-04-23T09:27:58.254916390Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-23T09:27:58.255383604Z INFO: Waiting for worker host service log collection to complete ...
2026-04-23T09:27:58.298163423Z INFO: Waiting for node performance related collection to complete ...
2026-04-23T09:27:58.803149731Z No resources found in openshift-etcd namespace.
2026-04-23T09:27:58.827892082Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-23T09:27:58.842789601Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-23T09:27:59.012823060Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-wsgmm
2026-04-23T09:27:59.012876891Z insights-runtime-extractor-g996f
2026-04-23T09:27:59.012895628Z insights-runtime-extractor-gk2qp
2026-04-23T09:27:59.012914525Z insights-runtime-extractor-wvl7x
2026-04-23T09:27:59.126603551Z Wrote inspect data to must-gather.
2026-04-23T09:27:59.160381485Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-23T09:27:59.160425234Z INFO: on-disk MachineConfig config collection complete.
2026-04-23T09:27:59.293467898Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-23T09:27:59.383187177Z No resources found
2026-04-23T09:27:59.419077801Z ERROR: No running kube-apiserver pods found
2026-04-23T09:27:59.559404629Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-23T09:27:59.559404629Z See 'oc rsync -h' for help and examples
2026-04-23T09:27:59.661549446Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-23T09:27:59.706552911Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-23T09:27:59.829962578Z INFO: "metallb-operator" not detected. Skipping.
2026-04-23T09:27:59.944811733Z Gathering data for ns/openshift-monitoring...
2026-04-23T09:27:59.980723581Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.013477866Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-23T09:28:00.148975285Z INFO: Worker host service log collection to complete.
2026-04-23T09:28:00.158736069Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-23T09:28:00.163192020Z No resources found
2026-04-23T09:28:00.173074131Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.250611745Z error: the server doesn't have a resource type "performanceprofile"
2026-04-23T09:28:00.278642916Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T09:28:00.315733837Z tar: Removing leading `/' from member names
2026-04-23T09:28:00.323539544Z INFO: HAProxy config collection complete.
2026-04-23T09:28:00.347920882Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.400825544Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.430443744Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.510111937Z INFO: Getting rules from prometheus-k8s-0
2026-04-23T09:28:00.522542212Z INFO: 'previous-logs' folder not found on ip-10-0-131-47.ec2.internal, skipping...
2026-04-23T09:28:00.573530739Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.670388595Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:00.698291704Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.731622852Z error: the server doesn't have a resource type "machineconfigs"
2026-04-23T09:28:00.788173805Z INFO: 'previous-logs' folder not found on ip-10-0-139-48.ec2.internal, skipping...
2026-04-23T09:28:00.941095251Z INFO: Getting status/config from prometheus-k8s-0
2026-04-23T09:28:00.961332829Z Wrote inspect data to must-gather.
2026-04-23T09:28:00.966503314Z Wrote inspect data to must-gather.
2026-04-23T09:28:01.056782591Z INFO: 'previous-logs' folder not found on ip-10-0-141-232.ec2.internal, skipping...
2026-04-23T09:28:01.056829127Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-23T09:28:01.056843722Z INFO: Machine Config Daemon termination log collection complete.
2026-04-23T09:28:01.181668435Z Wrote inspect data to must-gather.
2026-04-23T09:28:01.223721811Z Wrote inspect data to must-gather.
2026-04-23T09:28:01.345602258Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-23T09:28:01.355000604Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-23T09:28:01.499623025Z INFO: INTERCONNECT MODE
2026-04-23T09:28:01.499670441Z INFO: Gathering ovn-kubernetes DBs
2026-04-23T09:28:01.590222981Z Wrote inspect data to must-gather.
2026-04-23T09:28:01.598061644Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-23T09:28:01.644170036Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-23T09:28:01.681256222Z INFO: Gathering OVN_Northbound from ovnkube-node-9bsnx...
2026-04-23T09:28:01.681523944Z INFO: Gathering OVN_Northbound from ovnkube-node-9ltg5...
2026-04-23T09:28:01.681761374Z INFO: Gathering OVN_Northbound from ovnkube-node-k9zxl...
2026-04-23T09:28:01.682062095Z INFO: Gathering OVN_Southbound from ovnkube-node-9bsnx...
2026-04-23T09:28:01.682465384Z INFO: Gathering OVN_Southbound from ovnkube-node-9ltg5...
2026-04-23T09:28:01.682781173Z INFO: Gathering OVN_Southbound from ovnkube-node-k9zxl...
2026-04-23T09:28:01.941683079Z Wrote inspect data to must-gather.
2026-04-23T09:28:01.944543273Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-23T09:28:02.069596593Z tar: Removing leading `/' from member names
2026-04-23T09:28:02.085066481Z tar: Removing leading `/' from member names
2026-04-23T09:28:02.103056878Z tar: Removing leading `/' from member names
2026-04-23T09:28:02.107372066Z tar: Removing leading `/' from member names
2026-04-23T09:28:02.111908633Z tar: Removing leading `/' from member names
2026-04-23T09:28:02.114427972Z tar: Removing leading `/' from member names
2026-04-23T09:28:02.198512491Z Wrote inspect data to must-gather.
2026-04-23T09:28:02.234014411Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-23T09:28:02.411040142Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-23T09:28:02.521794035Z Wrote inspect data to must-gather.
2026-04-23T09:28:02.583039103Z INFO: Waiting for network log collection to complete ...
2026-04-23T09:28:02.583156160Z INFO: Waiting for ovnk database copies to complete ...
2026-04-23T09:28:02.583689931Z INFO: Copying ovnk databases complete.
2026-04-23T09:28:02.585988495Z 3.0M	must-gather/network_logs/ovnk_database_store
2026-04-23T09:28:02.589537468Z ovnk_database_store/
2026-04-23T09:28:02.589537468Z ovnk_database_store/ovnkube-node-9ltg5_nbdb
2026-04-23T09:28:02.600305337Z ovnk_database_store/ovnkube-node-k9zxl_nbdb
2026-04-23T09:28:02.610092552Z ovnk_database_store/ovnkube-node-9ltg5_sbdb
2026-04-23T09:28:02.618588388Z Gathering data for ns/openshift-network-console...
2026-04-23T09:28:02.627413811Z ovnk_database_store/ovnkube-node-k9zxl_sbdb
2026-04-23T09:28:02.648407087Z ovnk_database_store/ovnkube-node-9bsnx_sbdb
2026-04-23T09:28:02.656811723Z INFO: Getting status from alertmanager-main-0
2026-04-23T09:28:02.664814236Z ovnk_database_store/ovnkube-node-9bsnx_nbdb
2026-04-23T09:28:02.678534792Z INFO: Network log collection complete.
2026-04-23T09:28:02.816560775Z Wrote inspect data to must-gather.
2026-04-23T09:28:03.083005360Z Gathering data for ns/openshift-console-operator...
2026-04-23T09:28:03.090563391Z Wrote inspect data to must-gather.
2026-04-23T09:28:03.345071786Z Wrote inspect data to must-gather.
2026-04-23T09:28:03.473519274Z Gathering data for ns/openshift-console...
2026-04-23T09:28:03.595904503Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-23T09:28:03.599306470Z INFO: Fallback to identify the container image from release info
2026-04-23T09:28:03.916230292Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-23T09:28:04.133233566Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-23T09:28:04.262653475Z daemonset.apps/perf-node-gather-daemonset created
2026-04-23T09:28:04.322929299Z Gathering data for ns/openshift-dns-operator...
2026-04-23T09:28:04.400173624Z Waiting for performance profile collector pods to become ready: 1
2026-04-23T09:28:04.643378108Z Gathering data for ns/openshift-dns...
2026-04-23T09:28:05.375634920Z Gathering data for ns/openshift-image-registry...
2026-04-23T09:28:05.521098673Z Waiting for performance profile collector pods to become ready: 2
2026-04-23T09:28:05.675454020Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:05.943291304Z Gathering data for ns/openshift-ingress-operator...
2026-04-23T09:28:06.292476408Z Gathering data for ns/openshift-ingress...
2026-04-23T09:28:06.642080736Z Waiting for performance profile collector pods to become ready: 3
2026-04-23T09:28:06.702194299Z Gathering data for ns/openshift-ingress-canary...
2026-04-23T09:28:07.105429702Z Gathering data for ns/openshift-insights...
2026-04-23T09:28:07.767215423Z Waiting for performance profile collector pods to become ready: 4
2026-04-23T09:28:08.882094687Z Waiting for performance profile collector pods to become ready: 5
2026-04-23T09:28:08.959520923Z Gathering data for ns/openshift-jobset-operator...
2026-04-23T09:28:09.737384188Z Gathering data for ns/openshift-config...
2026-04-23T09:28:10.019538073Z Waiting for performance profile collector pods to become ready: 6
2026-04-23T09:28:10.038289563Z Gathering data for ns/openshift-config-managed...
2026-04-23T09:28:10.471441679Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-23T09:28:10.682259551Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:10.753435083Z Gathering data for ns/openshift-kube-apiserver...
2026-04-23T09:28:11.063056752Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-23T09:28:11.152658276Z Waiting for performance profile collector pods to become ready: 7
2026-04-23T09:28:11.412859800Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-23T09:28:11.767369131Z Gathering data for ns/openshift-kube-scheduler...
2026-04-23T09:28:12.063675129Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-23T09:28:12.264255355Z Waiting for performance profile collector pods to become ready: 8
2026-04-23T09:28:12.426910075Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-23T09:28:12.800012242Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-23T09:28:13.148434233Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-23T09:28:13.387238926Z Waiting for performance profile collector pods to become ready: 9
2026-04-23T09:28:13.684889147Z Gathering data for ns/openshift-multus...
2026-04-23T09:28:14.495212284Z Waiting for performance profile collector pods to become ready: 10
2026-04-23T09:28:15.617856393Z Waiting for performance profile collector pods to become ready: 11
2026-04-23T09:28:15.687445258Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:15.724971240Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-23T09:28:16.728897539Z Waiting for performance profile collector pods to become ready: 12
2026-04-23T09:28:17.553467910Z Gathering data for ns/openshift-host-network...
2026-04-23T09:28:17.849561194Z Waiting for performance profile collector pods to become ready: 13
2026-04-23T09:28:17.990065566Z Gathering data for ns/openshift-network-diagnostics...
2026-04-23T09:28:18.711718484Z Gathering data for ns/openshift-network-node-identity...
2026-04-23T09:28:18.970129663Z Waiting for performance profile collector pods to become ready: 14
2026-04-23T09:28:19.205857229Z Gathering data for ns/openshift-network-operator...
2026-04-23T09:28:19.758972622Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-23T09:28:20.100096085Z Waiting for performance profile collector pods to become ready: 15
2026-04-23T09:28:20.100332613Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-23T09:28:20.678991888Z Gathering data for ns/openshift-apiserver-operator...
2026-04-23T09:28:20.694151842Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:21.073626389Z Gathering data for ns/openshift-apiserver...
2026-04-23T09:28:21.221181283Z Waiting for performance profile collector pods to become ready: 16
2026-04-23T09:28:21.563871963Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-23T09:28:21.889015226Z Gathering data for ns/openshift-controller-manager...
2026-04-23T09:28:22.218770446Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-23T09:28:22.352175644Z Waiting for performance profile collector pods to become ready: 17
2026-04-23T09:28:22.886192759Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-23T09:28:23.205053444Z Gathering data for ns/openshift-service-ca-operator...
2026-04-23T09:28:23.477759432Z Waiting for performance profile collector pods to become ready: 18
2026-04-23T09:28:23.531748943Z Gathering data for ns/openshift-service-ca...
2026-04-23T09:28:23.892727943Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-23T09:28:24.600026028Z Waiting for performance profile collector pods to become ready: 19
2026-04-23T09:28:24.813359322Z Wrote inspect data to must-gather.
2026-04-23T09:28:24.813399351Z error: inspection completed with the errors occurred while gathering data:
2026-04-23T09:28:24.813399351Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-23T09:28:25.699290032Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:25.706826306Z Waiting for performance profile collector pods to become ready: 20
2026-04-23T09:28:26.812014861Z Waiting for performance profile collector pods to become ready: 21
2026-04-23T09:28:27.922787479Z Waiting for performance profile collector pods to become ready: 22
2026-04-23T09:28:29.027100008Z Waiting for performance profile collector pods to become ready: 23
2026-04-23T09:28:30.129208654Z Waiting for performance profile collector pods to become ready: 24
2026-04-23T09:28:30.704090471Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:31.234669290Z Waiting for performance profile collector pods to become ready: 25
2026-04-23T09:28:32.338717974Z Waiting for performance profile collector pods to become ready: 26
2026-04-23T09:28:33.447188543Z Waiting for performance profile collector pods to become ready: 27
2026-04-23T09:28:34.553206929Z Waiting for performance profile collector pods to become ready: 28
2026-04-23T09:28:35.659006677Z Waiting for performance profile collector pods to become ready: 29
2026-04-23T09:28:35.708742158Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:36.765902156Z Waiting for performance profile collector pods to become ready: 30
2026-04-23T09:28:37.869615367Z Waiting for performance profile collector pods to become ready: 31
2026-04-23T09:28:38.975602375Z Waiting for performance profile collector pods to become ready: 32
2026-04-23T09:28:40.077415394Z Waiting for performance profile collector pods to become ready: 33
2026-04-23T09:28:40.713443539Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:41.181091552Z Waiting for performance profile collector pods to become ready: 34
2026-04-23T09:28:42.291658887Z Waiting for performance profile collector pods to become ready: 35
2026-04-23T09:28:43.399917451Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-23T09:28:43.626213819Z Collecting performance related data for node ip-10-0-141-232.ec2.internal
2026-04-23T09:28:43.633511349Z Collecting performance related data for node ip-10-0-139-48.ec2.internal
2026-04-23T09:28:43.640769574Z Collecting performance related data for node ip-10-0-131-47.ec2.internal
2026-04-23T09:28:45.718173938Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:50.723435840Z [disk usage checker] Volume usage percentage: current = 42 ; allowed = 70
2026-04-23T09:28:51.443483122Z Collecting kubelet logs for node ip-10-0-141-232.ec2.internal
2026-04-23T09:28:51.445285088Z Collecting kubelet logs for node ip-10-0-139-48.ec2.internal
2026-04-23T09:28:51.447094540Z Collecting kubelet logs for node ip-10-0-131-47.ec2.internal
2026-04-23T09:28:51.765785884Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-23T09:28:51.771947499Z INFO: Node performance data collection complete.
2026-04-23T09:28:51.908798798Z error: the server doesn't have a resource type "clusters"
2026-04-23T09:28:51.982156567Z Caches written to disk