2026-05-25T06:26:09.140350099Z [disk usage checker] Started
2026-05-25T06:26:09.143561103Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-05-25T06:26:09.326528024Z Gathering data for ns/openshift-cluster-version...
2026-05-25T06:26:09.440871287Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T06:26:09.520705086Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T06:26:09.592225115Z Gathering data for ns/default...
2026-05-25T06:26:09.868610064Z Gathering data for ns/openshift...
2026-05-25T06:26:10.242926220Z Gathering data for ns/kube-system...
2026-05-25T06:26:10.893311169Z Gathering data for ns/openshift-etcd...
2026-05-25T06:26:11.162355669Z Wrote inspect data to must-gather.
2026-05-25T06:26:11.162384117Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T06:26:11.162384117Z     namespaces "assisted-installer" not found
2026-05-25T06:26:11.308963602Z Waiting on subprocesses to finish execution.
2026-05-25T06:26:11.317403137Z INFO: Gathering machine config daemon's old logs from all nodes
2026-05-25T06:26:11.320998636Z INFO: Collecting host service logs for crio
2026-05-25T06:26:11.321064722Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-05-25T06:26:11.321397003Z INFO: Collecting host service logs for kubelet
2026-05-25T06:26:11.321792614Z INFO: Collecting host service logs for rpm-ostreed
2026-05-25T06:26:11.322157222Z INFO: Collecting host service logs for ostree-finalize-staged
2026-05-25T06:26:11.322535857Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-05-25T06:26:11.322912452Z INFO: Collecting host service logs for machine-config-daemon-host
2026-05-25T06:26:11.323247299Z INFO: Collecting host service logs for NetworkManager
2026-05-25T06:26:11.323647098Z INFO: Collecting host service logs for openvswitch
2026-05-25T06:26:11.323992434Z INFO: Collecting host service logs for ovs-configuration
2026-05-25T06:26:11.324328127Z INFO: Collecting host service logs for ovsdb-server
2026-05-25T06:26:11.324670301Z INFO: Collecting host service logs for ovs-vswitchd
2026-05-25T06:26:11.324784273Z INFO: Gathering HAProxy config files
2026-05-25T06:26:11.325105655Z INFO: Waiting for worker host service log collection to complete ...
2026-05-25T06:26:11.331109078Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-05-25T06:26:11.375545975Z INFO: Waiting for node performance related collection to complete ...
2026-05-25T06:26:11.698400995Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-05-25T06:26:11.976193203Z No resources found
2026-05-25T06:26:12.042709390Z INFO: "sriov-network-operator" not detected. Skipping.
2026-05-25T06:26:12.068626101Z Wrote inspect data to must-gather.
2026-05-25T06:26:12.118946053Z INFO: Found 1 replicas - prometheus-k8s-0
2026-05-25T06:26:12.188829144Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-05-25T06:26:12.195120756Z INFO: "metallb-operator" not detected. Skipping.
2026-05-25T06:26:12.284397712Z No resources found in openshift-etcd namespace.
2026-05-25T06:26:12.356968343Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-05-25T06:26:12.357051203Z INFO: on-disk MachineConfig config collection complete.
2026-05-25T06:26:12.467108618Z INFO: Collecting Insights Archives from insights-operator-78bdfb888c-zk9p7
2026-05-25T06:26:12.467174132Z insights-runtime-extractor-6jxtw
2026-05-25T06:26:12.467194804Z insights-runtime-extractor-kdh8k
2026-05-25T06:26:12.467214098Z insights-runtime-extractor-zcxj8
2026-05-25T06:26:12.593390496Z INFO: Worker host service log collection to complete.
2026-05-25T06:26:12.610448290Z error: the server doesn't have a resource type "performanceprofile"
2026-05-25T06:26:12.892771607Z INFO: Waiting for HAProxy config collection to complete ...
2026-05-25T06:26:12.936206093Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.026509231Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.042996067Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-05-25T06:26:13.102141656Z No resources found
2026-05-25T06:26:13.227711263Z ERROR: No running kube-apiserver pods found
2026-05-25T06:26:13.296602771Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.345349516Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.345378884Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-05-25T06:26:13.345378884Z See 'oc rsync -h' for help and examples
2026-05-25T06:26:13.358822658Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-05-25T06:26:13.388569009Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T06:26:13.437356908Z INFO: 'previous-logs' folder not found on ip-10-0-128-119.ec2.internal, skipping...
2026-05-25T06:26:13.441120397Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.444221071Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.483696921Z tar: Removing leading `/' from member names
2026-05-25T06:26:13.488349342Z INFO: HAProxy config collection complete.
2026-05-25T06:26:13.599889933Z error: the server doesn't have a resource type "machineconfigs"
2026-05-25T06:26:13.636409450Z Gathering data for ns/openshift-monitoring...
2026-05-25T06:26:13.654649706Z INFO: Getting rules from prometheus-k8s-0
2026-05-25T06:26:13.700899392Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.712919046Z INFO: 'previous-logs' folder not found on ip-10-0-135-159.ec2.internal, skipping...
2026-05-25T06:26:13.817668209Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.943971019Z Wrote inspect data to must-gather.
2026-05-25T06:26:13.958912424Z INFO: 'previous-logs' folder not found on ip-10-0-143-2.ec2.internal, skipping...
2026-05-25T06:26:13.958982578Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-05-25T06:26:13.959053033Z INFO: Machine Config Daemon termination log collection complete.
2026-05-25T06:26:14.005175405Z Wrote inspect data to must-gather.
2026-05-25T06:26:14.062636181Z INFO: Getting status/config from prometheus-k8s-0
2026-05-25T06:26:14.149860485Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-05-25T06:26:14.153062532Z error: the server doesn't have a resource type "kubeletconfigs"
2026-05-25T06:26:14.175817975Z Wrote inspect data to must-gather.
2026-05-25T06:26:14.339659772Z Wrote inspect data to must-gather.
2026-05-25T06:26:14.420808833Z INFO: Getting status/flags from prometheus-k8s-0
2026-05-25T06:26:14.427310614Z INFO: INTERCONNECT MODE
2026-05-25T06:26:14.427334434Z INFO: Gathering ovn-kubernetes DBs
2026-05-25T06:26:14.533116300Z Wrote inspect data to must-gather.
2026-05-25T06:26:14.583842819Z INFO: Gathering OVN_Northbound from ovnkube-node-22rrb...
2026-05-25T06:26:14.584074023Z INFO: Gathering OVN_Northbound from ovnkube-node-pk6gn...
2026-05-25T06:26:14.584308512Z INFO: Gathering OVN_Northbound from ovnkube-node-sjlbb...
2026-05-25T06:26:14.584637866Z INFO: Gathering OVN_Southbound from ovnkube-node-22rrb...
2026-05-25T06:26:14.584905392Z INFO: Gathering OVN_Southbound from ovnkube-node-pk6gn...
2026-05-25T06:26:14.585140369Z INFO: Gathering OVN_Southbound from ovnkube-node-sjlbb...
2026-05-25T06:26:14.647052561Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-05-25T06:26:14.817803520Z Wrote inspect data to must-gather.
2026-05-25T06:26:14.895879751Z tar: Removing leading `/' from member names
2026-05-25T06:26:14.897213094Z tar: Removing leading `/' from member names
2026-05-25T06:26:14.897652921Z tar: Removing leading `/' from member names
2026-05-25T06:26:14.912053160Z tar: Removing leading `/' from member names
2026-05-25T06:26:14.925778832Z tar: Removing leading `/' from member names
2026-05-25T06:26:14.927754174Z tar: Removing leading `/' from member names
2026-05-25T06:26:14.973710308Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-05-25T06:26:15.000500610Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-05-25T06:26:15.059504040Z Wrote inspect data to must-gather.
2026-05-25T06:26:15.221367602Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-05-25T06:26:15.274504703Z Wrote inspect data to must-gather.
2026-05-25T06:26:15.304557486Z INFO: Waiting for network log collection to complete ...
2026-05-25T06:26:15.304619893Z INFO: Waiting for ovnk database copies to complete ...
2026-05-25T06:26:15.305096466Z INFO: Copying ovnk databases complete.
2026-05-25T06:26:15.307044393Z 3.7M	must-gather/network_logs/ovnk_database_store
2026-05-25T06:26:15.309783980Z ovnk_database_store/
2026-05-25T06:26:15.309783980Z ovnk_database_store/ovnkube-node-sjlbb_sbdb
2026-05-25T06:26:15.334271592Z ovnk_database_store/ovnkube-node-22rrb_nbdb
2026-05-25T06:26:15.342345927Z ovnk_database_store/ovnkube-node-sjlbb_nbdb
2026-05-25T06:26:15.350676949Z ovnk_database_store/ovnkube-node-22rrb_sbdb
2026-05-25T06:26:15.370419911Z ovnk_database_store/ovnkube-node-pk6gn_nbdb
2026-05-25T06:26:15.380435193Z ovnk_database_store/ovnkube-node-pk6gn_sbdb
2026-05-25T06:26:15.405318956Z INFO: Network log collection complete.
2026-05-25T06:26:15.454711779Z W0525 06:26:15.454674    1371 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-05-25T06:26:15.459687993Z Wrote inspect data to must-gather.
2026-05-25T06:26:15.560707398Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-05-25T06:26:15.570469935Z INFO: Getting status from alertmanager-main-0
2026-05-25T06:26:15.724726131Z Gathering data for ns/openshift-network-console...
2026-05-25T06:26:15.732646717Z Wrote inspect data to must-gather.
2026-05-25T06:26:15.967113069Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-05-25T06:26:15.970247251Z INFO: Fallback to identify the container image from release info
2026-05-25T06:26:16.091146703Z Gathering data for ns/openshift-console-operator...
2026-05-25T06:26:16.401426162Z Gathering data for ns/openshift-console...
2026-05-25T06:26:16.524878223Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:934b226bb305ecfe687cdb9684d7d4cf04b130e6254c2817dd1a6112d4829bda
2026-05-25T06:26:16.626753517Z daemonset.apps/perf-node-gather-daemonset created
2026-05-25T06:26:16.751364963Z Waiting for performance profile collector pods to become ready: 1
2026-05-25T06:26:16.797172726Z Gathering data for ns/openshift-cluster-storage-operator...
2026-05-25T06:26:17.126903805Z Gathering data for ns/openshift-dns-operator...
2026-05-25T06:26:17.409256127Z Gathering data for ns/openshift-dns...
2026-05-25T06:26:17.872783587Z Waiting for performance profile collector pods to become ready: 2
2026-05-25T06:26:17.930045629Z Gathering data for ns/openshift-image-registry...
2026-05-25T06:26:18.354712789Z Gathering data for ns/openshift-ingress-operator...
2026-05-25T06:26:18.642451124Z Gathering data for ns/openshift-ingress...
2026-05-25T06:26:18.927776777Z Gathering data for ns/openshift-ingress-canary...
2026-05-25T06:26:18.988221748Z Waiting for performance profile collector pods to become ready: 3
2026-05-25T06:26:19.154744847Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-05-25T06:26:19.273055317Z Gathering data for ns/openshift-insights...
2026-05-25T06:26:20.096090478Z Waiting for performance profile collector pods to become ready: 4
2026-05-25T06:26:21.138057742Z Gathering data for ns/openshift-config...
2026-05-25T06:26:21.214379971Z Waiting for performance profile collector pods to become ready: 5
2026-05-25T06:26:21.378689930Z Gathering data for ns/openshift-config-managed...
2026-05-25T06:26:21.756933176Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-05-25T06:26:21.994742086Z Gathering data for ns/openshift-kube-apiserver...
2026-05-25T06:26:22.220144405Z Gathering data for ns/openshift-kube-controller-manager...
2026-05-25T06:26:22.332980082Z Waiting for performance profile collector pods to become ready: 6
2026-05-25T06:26:22.518650619Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-05-25T06:26:22.776370481Z Gathering data for ns/openshift-kube-scheduler...
2026-05-25T06:26:23.008229540Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-05-25T06:26:23.267034325Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-05-25T06:26:23.439334817Z Waiting for performance profile collector pods to become ready: 7
2026-05-25T06:26:23.569733234Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-05-25T06:26:23.837424023Z Gathering data for ns/openshift-user-workload-monitoring...
2026-05-25T06:26:24.160025966Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T06:26:24.308677447Z Gathering data for ns/openshift-multus...
2026-05-25T06:26:24.537833812Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-05-25T06:26:24.753766924Z Collecting performance related data for node ip-10-0-143-2.ec2.internal
2026-05-25T06:26:24.760653549Z Collecting performance related data for node ip-10-0-128-119.ec2.internal
2026-05-25T06:26:24.768294923Z Collecting performance related data for node ip-10-0-135-159.ec2.internal
2026-05-25T06:26:25.713040886Z Gathering data for ns/openshift-ovn-kubernetes...
2026-05-25T06:26:27.217827081Z Gathering data for ns/openshift-host-network...
2026-05-25T06:26:27.570825237Z Gathering data for ns/openshift-network-diagnostics...
2026-05-25T06:26:28.080353922Z Gathering data for ns/openshift-network-node-identity...
2026-05-25T06:26:28.433781665Z Gathering data for ns/openshift-network-operator...
2026-05-25T06:26:28.767631505Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-05-25T06:26:29.005656412Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-05-25T06:26:29.164510638Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-05-25T06:26:29.411448383Z Gathering data for ns/openshift-apiserver-operator...
2026-05-25T06:26:29.684097153Z Gathering data for ns/openshift-apiserver...
2026-05-25T06:26:30.036020994Z Gathering data for ns/openshift-controller-manager-operator...
2026-05-25T06:26:30.278208901Z Gathering data for ns/openshift-controller-manager...
2026-05-25T06:26:30.521551212Z Gathering data for ns/openshift-cluster-samples-operator...
2026-05-25T06:26:31.106580552Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-05-25T06:26:31.366253807Z Gathering data for ns/openshift-service-ca-operator...
2026-05-25T06:26:31.638483983Z Gathering data for ns/openshift-service-ca...
2026-05-25T06:26:32.051771112Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-05-25T06:26:32.355423187Z Collecting kubelet logs for node ip-10-0-143-2.ec2.internal
2026-05-25T06:26:32.357179474Z Collecting kubelet logs for node ip-10-0-128-119.ec2.internal
2026-05-25T06:26:32.359247503Z Collecting kubelet logs for node ip-10-0-135-159.ec2.internal
2026-05-25T06:26:32.703065019Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-05-25T06:26:32.708320664Z INFO: Node performance data collection complete.
2026-05-25T06:26:32.949229856Z Wrote inspect data to must-gather.
2026-05-25T06:26:32.949263086Z error: inspection completed with the errors occurred while gathering data:
2026-05-25T06:26:32.949263086Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-05-25T06:26:33.071186304Z error: the server doesn't have a resource type "clusters"
2026-05-25T06:26:33.201868037Z Caches written to disk