2026-04-22T20:11:53.703214842Z [disk usage checker] Started
2026-04-22T20:11:53.706733252Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-22T20:11:53.896874007Z Gathering data for ns/openshift-cluster-version...
2026-04-22T20:11:54.015078725Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T20:11:54.088827171Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T20:11:54.162391011Z Gathering data for ns/default...
2026-04-22T20:11:54.446224877Z Gathering data for ns/openshift...
2026-04-22T20:11:54.826919000Z Gathering data for ns/kube-system...
2026-04-22T20:11:55.457481409Z Gathering data for ns/openshift-etcd...
2026-04-22T20:11:55.755524886Z Wrote inspect data to must-gather.
2026-04-22T20:11:55.755561077Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T20:11:55.755561077Z     namespaces "assisted-installer" not found
2026-04-22T20:11:56.071972134Z Waiting on subprocesses to finish execution.
2026-04-22T20:11:56.077198335Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-22T20:11:56.083340508Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-22T20:11:56.090885142Z INFO: Collecting host service logs for crio
2026-04-22T20:11:56.091233242Z INFO: Collecting host service logs for kubelet
2026-04-22T20:11:56.091514003Z INFO: Gathering HAProxy config files
2026-04-22T20:11:56.091571616Z INFO: Collecting host service logs for rpm-ostreed
2026-04-22T20:11:56.091888446Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-22T20:11:56.092251918Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-22T20:11:56.092598196Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-22T20:11:56.092933338Z INFO: Collecting host service logs for NetworkManager
2026-04-22T20:11:56.093251622Z INFO: Collecting host service logs for openvswitch
2026-04-22T20:11:56.093581618Z INFO: Collecting host service logs for ovs-configuration
2026-04-22T20:11:56.094057321Z INFO: Collecting host service logs for ovsdb-server
2026-04-22T20:11:56.094388838Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-22T20:11:56.094848040Z INFO: Waiting for worker host service log collection to complete ...
2026-04-22T20:11:56.100468787Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-22T20:11:56.172588470Z INFO: Waiting for node performance related collection to complete ...
2026-04-22T20:11:56.738724837Z No resources found in openshift-etcd namespace.
2026-04-22T20:11:56.793668776Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-22T20:11:56.842585856Z No resources found
2026-04-22T20:11:56.921215392Z Wrote inspect data to must-gather.
2026-04-22T20:11:57.028130871Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-22T20:11:57.152093200Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-22T20:11:57.152169687Z INFO: on-disk MachineConfig config collection complete.
2026-04-22T20:11:57.366621586Z INFO: "metallb-operator" not detected. Skipping.
2026-04-22T20:11:57.369321638Z INFO: Worker host service log collection to complete.
2026-04-22T20:11:57.380506571Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-22T20:11:57.398899131Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-748dj
2026-04-22T20:11:57.398899131Z insights-runtime-extractor-9t2qh
2026-04-22T20:11:57.398899131Z insights-runtime-extractor-h762s
2026-04-22T20:11:57.398899131Z insights-runtime-extractor-nm6zf
2026-04-22T20:11:57.422490973Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-22T20:11:57.517644335Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-22T20:11:57.523293184Z error: the server doesn't have a resource type "performanceprofile"
2026-04-22T20:11:57.866770029Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.024141767Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-22T20:11:58.043082139Z ERROR: No running kube-apiserver pods found
2026-04-22T20:11:58.151161181Z No resources found
2026-04-22T20:11:58.178327529Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T20:11:58.216347633Z Gathering data for ns/openshift-monitoring...
2026-04-22T20:11:58.223583307Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-22T20:11:58.229062345Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-22T20:11:58.229062345Z See 'oc rsync -h' for help and examples
2026-04-22T20:11:58.344350077Z INFO: HAProxy config collection complete.
2026-04-22T20:11:58.375787467Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.384520299Z INFO: 'previous-logs' folder not found on ip-10-0-129-145.ec2.internal, skipping...
2026-04-22T20:11:58.391399904Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.409756397Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.461076757Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.465997771Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.592986103Z error: the server doesn't have a resource type "machineconfigs"
2026-04-22T20:11:58.666111821Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.675654241Z INFO: 'previous-logs' folder not found on ip-10-0-129-175.ec2.internal, skipping...
2026-04-22T20:11:58.714641341Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-22T20:11:58.753176452Z INFO: Getting rules from prometheus-k8s-0
2026-04-22T20:11:58.823568606Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.929396664Z Wrote inspect data to must-gather.
2026-04-22T20:11:58.951666036Z INFO: 'previous-logs' folder not found on ip-10-0-143-198.ec2.internal, skipping...
2026-04-22T20:11:58.951766268Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-22T20:11:58.952078309Z INFO: Machine Config Daemon termination log collection complete.
2026-04-22T20:11:59.047471446Z Wrote inspect data to must-gather.
2026-04-22T20:11:59.170558907Z INFO: Getting status/config from prometheus-k8s-0
2026-04-22T20:11:59.182328199Z Wrote inspect data to must-gather.
2026-04-22T20:11:59.209321573Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-22T20:11:59.429900071Z Wrote inspect data to must-gather.
2026-04-22T20:11:59.499487610Z INFO: INTERCONNECT MODE
2026-04-22T20:11:59.499529537Z INFO: Gathering ovn-kubernetes DBs
2026-04-22T20:11:59.530714422Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-22T20:11:59.589841737Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-22T20:11:59.642100772Z Wrote inspect data to must-gather.
2026-04-22T20:11:59.665853213Z INFO: Gathering OVN_Northbound from ovnkube-node-r2gjj...
2026-04-22T20:11:59.666106504Z INFO: Gathering OVN_Northbound from ovnkube-node-rsnsl...
2026-04-22T20:11:59.666372758Z INFO: Gathering OVN_Northbound from ovnkube-node-vrxcm...
2026-04-22T20:11:59.666630934Z INFO: Gathering OVN_Southbound from ovnkube-node-r2gjj...
2026-04-22T20:11:59.666898773Z INFO: Gathering OVN_Southbound from ovnkube-node-rsnsl...
2026-04-22T20:11:59.667185361Z INFO: Gathering OVN_Southbound from ovnkube-node-vrxcm...
2026-04-22T20:11:59.897366838Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-22T20:11:59.961547526Z tar: Removing leading `/' from member names
2026-04-22T20:11:59.972088672Z Wrote inspect data to must-gather.
2026-04-22T20:11:59.987588662Z tar: Removing leading `/' from member names
2026-04-22T20:11:59.991037103Z tar: Removing leading `/' from member names
2026-04-22T20:11:59.996841819Z tar: Removing leading `/' from member names
2026-04-22T20:12:00.003654347Z tar: Removing leading `/' from member names
2026-04-22T20:12:00.010274402Z tar: Removing leading `/' from member names
2026-04-22T20:12:00.168104929Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-22T20:12:00.202378760Z W0422 20:12:00.202327    1281 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-04-22T20:12:00.209678100Z Wrote inspect data to must-gather.
2026-04-22T20:12:00.235397510Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-22T20:12:00.418838490Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-22T20:12:00.439254126Z Gathering data for ns/openshift-network-console...
2026-04-22T20:12:00.539400772Z INFO: Waiting for network log collection to complete ...
2026-04-22T20:12:00.539518322Z INFO: Waiting for ovnk database copies to complete ...
2026-04-22T20:12:00.539680231Z Wrote inspect data to must-gather.
2026-04-22T20:12:00.540056366Z INFO: Copying ovnk databases complete.
2026-04-22T20:12:00.542847196Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-04-22T20:12:00.545650510Z ovnk_database_store/
2026-04-22T20:12:00.545716106Z ovnk_database_store/ovnkube-node-rsnsl_nbdb
2026-04-22T20:12:00.558418652Z ovnk_database_store/ovnkube-node-r2gjj_sbdb
2026-04-22T20:12:00.578644217Z ovnk_database_store/ovnkube-node-rsnsl_sbdb
2026-04-22T20:12:00.602185708Z ovnk_database_store/ovnkube-node-vrxcm_nbdb
2026-04-22T20:12:00.614005775Z ovnk_database_store/ovnkube-node-r2gjj_nbdb
2026-04-22T20:12:00.623954806Z ovnk_database_store/ovnkube-node-vrxcm_sbdb
2026-04-22T20:12:00.652963714Z INFO: Network log collection complete.
2026-04-22T20:12:00.814525209Z INFO: Getting status from alertmanager-main-0
2026-04-22T20:12:00.835063221Z Wrote inspect data to must-gather.
2026-04-22T20:12:00.886268358Z Gathering data for ns/openshift-console-operator...
2026-04-22T20:12:01.114267224Z Wrote inspect data to must-gather.
2026-04-22T20:12:01.241294201Z Gathering data for ns/openshift-console...
2026-04-22T20:12:01.378477181Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-22T20:12:01.380594449Z INFO: Fallback to identify the container image from release info
2026-04-22T20:12:01.655373510Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-22T20:12:01.953731726Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-22T20:12:01.958425982Z Gathering data for ns/openshift-dns-operator...
2026-04-22T20:12:02.084344173Z daemonset.apps/perf-node-gather-daemonset created
2026-04-22T20:12:02.220832650Z Waiting for performance profile collector pods to become ready: 1
2026-04-22T20:12:02.231399080Z Gathering data for ns/openshift-dns...
2026-04-22T20:12:02.757961367Z Gathering data for ns/openshift-image-registry...
2026-04-22T20:12:03.214447180Z Gathering data for ns/openshift-ingress-operator...
2026-04-22T20:12:03.347087801Z Waiting for performance profile collector pods to become ready: 2
2026-04-22T20:12:03.509944559Z Gathering data for ns/openshift-ingress...
2026-04-22T20:12:03.720197178Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-22T20:12:03.784330390Z Gathering data for ns/openshift-ingress-canary...
2026-04-22T20:12:04.102158651Z Gathering data for ns/openshift-insights...
2026-04-22T20:12:04.468433983Z Waiting for performance profile collector pods to become ready: 3
2026-04-22T20:12:05.583058885Z Waiting for performance profile collector pods to become ready: 4
2026-04-22T20:12:06.106809770Z Gathering data for ns/kserve...
2026-04-22T20:12:06.693737908Z Waiting for performance profile collector pods to become ready: 5
2026-04-22T20:12:07.809478661Z Waiting for performance profile collector pods to become ready: 6
2026-04-22T20:12:07.834910025Z Gathering data for ns/openshift-config...
2026-04-22T20:12:08.099141992Z Gathering data for ns/openshift-config-managed...
2026-04-22T20:12:08.471174080Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-22T20:12:08.691675674Z Gathering data for ns/openshift-kube-apiserver...
2026-04-22T20:12:08.725484629Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-22T20:12:08.930792207Z Waiting for performance profile collector pods to become ready: 7
2026-04-22T20:12:08.962624414Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-22T20:12:09.206118433Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-22T20:12:09.493798243Z Gathering data for ns/openshift-kube-scheduler...
2026-04-22T20:12:09.713313292Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-22T20:12:09.954722491Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-22T20:12:10.047668247Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-22T20:12:10.263364104Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-22T20:12:10.311262094Z Collecting performance related data for node ip-10-0-129-175.ec2.internal
2026-04-22T20:12:10.321791803Z Collecting performance related data for node ip-10-0-129-145.ec2.internal
2026-04-22T20:12:10.329845050Z Collecting performance related data for node ip-10-0-143-198.ec2.internal
2026-04-22T20:12:10.588411907Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-22T20:12:11.160366458Z Gathering data for ns/openshift-multus...
2026-04-22T20:12:12.392636466Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-22T20:12:13.731986501Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-22T20:12:13.753356737Z Gathering data for ns/openshift-host-network...
2026-04-22T20:12:14.016823567Z Gathering data for ns/openshift-network-diagnostics...
2026-04-22T20:12:14.476520827Z Gathering data for ns/openshift-network-node-identity...
2026-04-22T20:12:14.836741584Z Gathering data for ns/openshift-network-operator...
2026-04-22T20:12:15.162103316Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-22T20:12:15.392442301Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-22T20:12:15.794994012Z Gathering data for ns/openshift-apiserver-operator...
2026-04-22T20:12:16.048818634Z Gathering data for ns/openshift-apiserver...
2026-04-22T20:12:16.430673093Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-22T20:12:16.650156358Z Gathering data for ns/openshift-controller-manager...
2026-04-22T20:12:16.928626621Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-22T20:12:17.559513596Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-22T20:12:17.821951395Z Gathering data for ns/openshift-service-ca-operator...
2026-04-22T20:12:18.079019985Z Gathering data for ns/openshift-service-ca...
2026-04-22T20:12:18.094961051Z Collecting kubelet logs for node ip-10-0-129-175.ec2.internal
2026-04-22T20:12:18.096721355Z Collecting kubelet logs for node ip-10-0-129-145.ec2.internal
2026-04-22T20:12:18.098285689Z Collecting kubelet logs for node ip-10-0-143-198.ec2.internal
2026-04-22T20:12:18.463014631Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-22T20:12:18.472171801Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-22T20:12:18.477981490Z INFO: Node performance data collection complete.
2026-04-22T20:12:18.736903458Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-04-22T20:12:19.261127792Z Wrote inspect data to must-gather.
2026-04-22T20:12:19.261176182Z error: inspection completed with the errors occurred while gathering data:
2026-04-22T20:12:19.261176182Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-22T20:12:19.390413381Z error: the server doesn't have a resource type "clusters"
2026-04-22T20:12:19.584303158Z Caches written to disk