2026-04-16T22:36:36.823720881Z [disk usage checker] Started
2026-04-16T22:36:36.826831298Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T22:36:37.029058299Z Gathering data for ns/openshift-cluster-version...
2026-04-16T22:36:37.155534265Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T22:36:37.234089088Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:36:37.308949794Z Gathering data for ns/default...
2026-04-16T22:36:37.596103531Z Gathering data for ns/openshift...
2026-04-16T22:36:37.992779227Z Gathering data for ns/kube-system...
2026-04-16T22:36:38.657033569Z Gathering data for ns/openshift-etcd...
2026-04-16T22:36:38.947557377Z Wrote inspect data to must-gather.
2026-04-16T22:36:38.947589099Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T22:36:38.947589099Z     namespaces "assisted-installer" not found
2026-04-16T22:36:39.246143462Z Waiting on subprocesses to finish execution.
2026-04-16T22:36:39.257503466Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-16T22:36:39.260239359Z INFO: Gathering HAProxy config files
2026-04-16T22:36:39.261957916Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-16T22:36:39.275655800Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-16T22:36:39.291670646Z INFO: Collecting host service logs for crio
2026-04-16T22:36:39.292103258Z INFO: Collecting host service logs for kubelet
2026-04-16T22:36:39.292461638Z INFO: Collecting host service logs for rpm-ostreed
2026-04-16T22:36:39.292779767Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-16T22:36:39.293132174Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-16T22:36:39.293456544Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-16T22:36:39.293759259Z INFO: Collecting host service logs for NetworkManager
2026-04-16T22:36:39.294083950Z INFO: Collecting host service logs for openvswitch
2026-04-16T22:36:39.294405611Z INFO: Collecting host service logs for ovs-configuration
2026-04-16T22:36:39.294757000Z INFO: Collecting host service logs for ovsdb-server
2026-04-16T22:36:39.295114000Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-16T22:36:39.295484475Z INFO: Waiting for worker host service log collection to complete ...
2026-04-16T22:36:39.334863078Z INFO: Waiting for node performance related collection to complete ...
2026-04-16T22:36:39.855063152Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-16T22:36:39.858171716Z INFO: on-disk MachineConfig config collection complete.
2026-04-16T22:36:40.080101232Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-nf8x4
2026-04-16T22:36:40.080216147Z insights-runtime-extractor-j42tt
2026-04-16T22:36:40.080236830Z insights-runtime-extractor-lmj7s
2026-04-16T22:36:40.080255813Z insights-runtime-extractor-rspvv
2026-04-16T22:36:40.237930274Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-16T22:36:40.240781691Z No resources found in openshift-etcd namespace.
2026-04-16T22:36:40.243315488Z Wrote inspect data to must-gather.
2026-04-16T22:36:40.420059849Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-16T22:36:40.550604267Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-16T22:36:40.613399169Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-16T22:36:40.809498534Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-16T22:36:40.911071558Z INFO: "metallb-operator" not detected. Skipping.
2026-04-16T22:36:40.915576114Z No resources found
2026-04-16T22:36:41.006323032Z error: the server doesn't have a resource type "performanceprofile"
2026-04-16T22:36:41.045986428Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-16T22:36:41.045986428Z See 'oc rsync -h' for help and examples
2026-04-16T22:36:41.053207321Z INFO: Worker host service log collection to complete.
2026-04-16T22:36:41.173861522Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.184837441Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:36:41.369689413Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-16T22:36:41.375756179Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-16T22:36:41.384836872Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.439913317Z No resources found
2026-04-16T22:36:41.457544446Z ERROR: No running kube-apiserver pods found
2026-04-16T22:36:41.541310273Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.542214080Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.574675865Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.620254880Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.682598047Z INFO: 'previous-logs' folder not found on ip-10-0-129-102.ec2.internal, skipping...
2026-04-16T22:36:41.687708809Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.689412135Z Gathering data for ns/openshift-monitoring...
2026-04-16T22:36:41.698438150Z tar: Removing leading `/' from member names
2026-04-16T22:36:41.706444357Z INFO: HAProxy config collection complete.
2026-04-16T22:36:41.803008933Z error: the server doesn't have a resource type "machineconfigs"
2026-04-16T22:36:41.835454144Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T22:36:41.909748010Z INFO: Getting rules from prometheus-k8s-0
2026-04-16T22:36:41.972530981Z Wrote inspect data to must-gather.
2026-04-16T22:36:41.972659211Z INFO: 'previous-logs' folder not found on ip-10-0-140-65.ec2.internal, skipping...
2026-04-16T22:36:42.037854097Z Wrote inspect data to must-gather.
2026-04-16T22:36:42.228066523Z Wrote inspect data to must-gather.
2026-04-16T22:36:42.235071593Z INFO: 'previous-logs' folder not found on ip-10-0-142-35.ec2.internal, skipping...
2026-04-16T22:36:42.235071593Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-16T22:36:42.235106999Z INFO: Machine Config Daemon termination log collection complete.
2026-04-16T22:36:42.296169562Z Wrote inspect data to must-gather.
2026-04-16T22:36:42.325132567Z INFO: Getting status/config from prometheus-k8s-0
2026-04-16T22:36:42.452413533Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-16T22:36:42.531860458Z INFO: INTERCONNECT MODE
2026-04-16T22:36:42.532004491Z INFO: Gathering ovn-kubernetes DBs
2026-04-16T22:36:42.663682917Z Wrote inspect data to must-gather.
2026-04-16T22:36:42.699350752Z INFO: Gathering OVN_Northbound from ovnkube-node-5ggql...
2026-04-16T22:36:42.699568662Z INFO: Gathering OVN_Northbound from ovnkube-node-8trxs...
2026-04-16T22:36:42.699741528Z INFO: Gathering OVN_Northbound from ovnkube-node-qntcx...
2026-04-16T22:36:42.700003427Z INFO: Gathering OVN_Southbound from ovnkube-node-5ggql...
2026-04-16T22:36:42.700265201Z INFO: Gathering OVN_Southbound from ovnkube-node-8trxs...
2026-04-16T22:36:42.700543742Z INFO: Gathering OVN_Southbound from ovnkube-node-qntcx...
2026-04-16T22:36:42.780693959Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-16T22:36:42.987135483Z tar: Removing leading `/' from member names
2026-04-16T22:36:43.012476280Z Wrote inspect data to must-gather.
2026-04-16T22:36:43.020156718Z tar: Removing leading `/' from member names
2026-04-16T22:36:43.023681568Z tar: Removing leading `/' from member names
2026-04-16T22:36:43.051123593Z tar: Removing leading `/' from member names
2026-04-16T22:36:43.053352359Z tar: Removing leading `/' from member names
2026-04-16T22:36:43.056913632Z tar: Removing leading `/' from member names
2026-04-16T22:36:43.114427977Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-16T22:36:43.155112657Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-16T22:36:43.249065345Z Wrote inspect data to must-gather.
2026-04-16T22:36:43.414928936Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-16T22:36:43.489477347Z Wrote inspect data to must-gather.
2026-04-16T22:36:43.555686357Z INFO: Waiting for network log collection to complete ...
2026-04-16T22:36:43.555811262Z INFO: Waiting for ovnk database copies to complete ...
2026-04-16T22:36:43.556366288Z INFO: Copying ovnk databases complete.
2026-04-16T22:36:43.559093163Z 3.3M	must-gather/network_logs/ovnk_database_store
2026-04-16T22:36:43.563080480Z ovnk_database_store/
2026-04-16T22:36:43.563080480Z ovnk_database_store/ovnkube-node-qntcx_nbdb
2026-04-16T22:36:43.575451222Z ovnk_database_store/ovnkube-node-5ggql_sbdb
2026-04-16T22:36:43.594272921Z ovnk_database_store/ovnkube-node-8trxs_nbdb
2026-04-16T22:36:43.604519435Z ovnk_database_store/ovnkube-node-5ggql_nbdb
2026-04-16T22:36:43.611078694Z ovnk_database_store/ovnkube-node-qntcx_sbdb
2026-04-16T22:36:43.629802022Z ovnk_database_store/ovnkube-node-8trxs_sbdb
2026-04-16T22:36:43.654826857Z INFO: Network log collection complete.
2026-04-16T22:36:43.667498446Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-16T22:36:43.704203886Z W0416 22:36:43.704155    1370 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-16T22:36:43.709669147Z Wrote inspect data to must-gather.
2026-04-16T22:36:43.747920294Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-16T22:36:43.917625376Z Gathering data for ns/openshift-network-console...
2026-04-16T22:36:44.006264824Z Wrote inspect data to must-gather.
2026-04-16T22:36:44.060955635Z INFO: Getting status from alertmanager-main-0
2026-04-16T22:36:44.318633535Z Wrote inspect data to must-gather.
2026-04-16T22:36:44.341194563Z Gathering data for ns/openshift-console-operator...
2026-04-16T22:36:44.572749916Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-16T22:36:44.577222392Z INFO: Fallback to identify the container image from release info
2026-04-16T22:36:44.693979349Z Gathering data for ns/openshift-console...
2026-04-16T22:36:45.095280102Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-16T22:36:45.131283105Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-16T22:36:45.262047505Z daemonset.apps/perf-node-gather-daemonset created
2026-04-16T22:36:45.402922752Z Waiting for performance profile collector pods to become ready: 1
2026-04-16T22:36:45.451979219Z Gathering data for ns/openshift-dns-operator...
2026-04-16T22:36:45.745906760Z Gathering data for ns/openshift-dns...
2026-04-16T22:36:46.300944127Z Gathering data for ns/openshift-image-registry...
2026-04-16T22:36:46.536804135Z Waiting for performance profile collector pods to become ready: 2
2026-04-16T22:36:46.754233979Z Gathering data for ns/openshift-ingress-operator...
2026-04-16T22:36:46.840092957Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T22:36:47.058384629Z Gathering data for ns/openshift-ingress...
2026-04-16T22:36:47.367641473Z Gathering data for ns/openshift-ingress-canary...
2026-04-16T22:36:47.667289963Z Waiting for performance profile collector pods to become ready: 3
2026-04-16T22:36:47.735697610Z Gathering data for ns/openshift-insights...
2026-04-16T22:36:48.796109004Z Waiting for performance profile collector pods to become ready: 4
2026-04-16T22:36:49.858447426Z Gathering data for ns/kserve...
2026-04-16T22:36:49.912587499Z Waiting for performance profile collector pods to become ready: 5
2026-04-16T22:36:51.024787220Z Waiting for performance profile collector pods to become ready: 6
2026-04-16T22:36:51.259019981Z Gathering data for ns/openshift-config...
2026-04-16T22:36:51.513321382Z Gathering data for ns/openshift-config-managed...
2026-04-16T22:36:51.845349014Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T22:36:51.904547821Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-16T22:36:52.163499140Z Gathering data for ns/openshift-kube-apiserver...
2026-04-16T22:36:52.164314033Z Waiting for performance profile collector pods to become ready: 7
2026-04-16T22:36:52.457264641Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-16T22:36:52.732835578Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-16T22:36:53.006645691Z Gathering data for ns/openshift-kube-scheduler...
2026-04-16T22:36:53.267464208Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-16T22:36:53.301536141Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-16T22:36:53.567188956Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-16T22:36:53.603890781Z Collecting performance related data for node ip-10-0-140-65.ec2.internal
2026-04-16T22:36:53.610797890Z Collecting performance related data for node ip-10-0-142-35.ec2.internal
2026-04-16T22:36:53.618176520Z Collecting performance related data for node ip-10-0-129-102.ec2.internal
2026-04-16T22:36:53.957689752Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-16T22:36:54.340409202Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-16T22:36:54.974476736Z Gathering data for ns/openshift-multus...
2026-04-16T22:36:56.359749590Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-16T22:36:56.850197195Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T22:36:57.757912034Z Gathering data for ns/openshift-host-network...
2026-04-16T22:36:58.111379769Z Gathering data for ns/openshift-network-diagnostics...
2026-04-16T22:36:58.616907763Z Gathering data for ns/openshift-network-node-identity...
2026-04-16T22:36:59.010746376Z Gathering data for ns/openshift-network-operator...
2026-04-16T22:36:59.367285151Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-16T22:36:59.619524395Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-16T22:37:00.046730242Z Gathering data for ns/openshift-apiserver-operator...
2026-04-16T22:37:00.326700817Z Gathering data for ns/openshift-apiserver...
2026-04-16T22:37:00.718483974Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-16T22:37:00.976767204Z Gathering data for ns/openshift-controller-manager...
2026-04-16T22:37:01.248594481Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-16T22:37:01.638645558Z Collecting kubelet logs for node ip-10-0-140-65.ec2.internal
2026-04-16T22:37:01.640433613Z Collecting kubelet logs for node ip-10-0-142-35.ec2.internal
2026-04-16T22:37:01.642096034Z Collecting kubelet logs for node ip-10-0-129-102.ec2.internal
2026-04-16T22:37:01.855765036Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-04-16T22:37:01.915618473Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-16T22:37:02.019519353Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-16T22:37:02.024148060Z INFO: Node performance data collection complete.
2026-04-16T22:37:02.207103344Z Gathering data for ns/openshift-service-ca-operator...
2026-04-16T22:37:02.508573651Z Gathering data for ns/openshift-service-ca...
2026-04-16T22:37:02.855247448Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-16T22:37:03.702862823Z Wrote inspect data to must-gather.
2026-04-16T22:37:03.702929130Z error: inspection completed with the errors occurred while gathering data:
2026-04-16T22:37:03.702929130Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-16T22:37:03.833520334Z error: the server doesn't have a resource type "clusters"
2026-04-16T22:37:03.975168467Z Caches written to disk