2026-04-28T20:08:42.027265684Z [disk usage checker] Started
2026-04-28T20:08:42.066651184Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-28T20:08:42.256874621Z Gathering data for ns/openshift-cluster-version...
2026-04-28T20:08:42.400405682Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-28T20:08:42.497489222Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-28T20:08:42.578693449Z Gathering data for ns/default...
2026-04-28T20:08:42.925391096Z Gathering data for ns/openshift...
2026-04-28T20:08:43.348441973Z Gathering data for ns/kube-system...
2026-04-28T20:08:44.110030548Z Gathering data for ns/openshift-etcd...
2026-04-28T20:08:44.419477503Z Wrote inspect data to must-gather.
2026-04-28T20:08:44.419512351Z error: inspection completed with the errors occurred while gathering data:
2026-04-28T20:08:44.419512351Z     namespaces "assisted-installer" not found
2026-04-28T20:08:44.506918133Z Waiting on subprocesses to finish execution.
2026-04-28T20:08:44.512433460Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-04-28T20:08:44.514056809Z INFO: Collecting host service logs for crio
2026-04-28T20:08:44.514371708Z INFO: Collecting host service logs for kubelet
2026-04-28T20:08:44.514804493Z INFO: Collecting host service logs for rpm-ostreed
2026-04-28T20:08:44.515135977Z INFO: Collecting host service logs for ostree-finalize-staged
2026-04-28T20:08:44.515466195Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-04-28T20:08:44.515817445Z INFO: Collecting host service logs for machine-config-daemon-host
2026-04-28T20:08:44.516229119Z INFO: Collecting host service logs for NetworkManager
2026-04-28T20:08:44.516566548Z INFO: Collecting host service logs for openvswitch
2026-04-28T20:08:44.516915436Z INFO: Collecting host service logs for ovs-configuration
2026-04-28T20:08:44.517249055Z INFO: Collecting host service logs for ovsdb-server
2026-04-28T20:08:44.517595734Z INFO: Collecting host service logs for ovs-vswitchd
2026-04-28T20:08:44.518025647Z INFO: Waiting for worker host service log collection to complete ...
2026-04-28T20:08:44.543509959Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-04-28T20:08:44.555509501Z INFO: Gathering machine config daemon's old logs from all nodes
2026-04-28T20:08:44.591757039Z INFO: Gathering HAProxy config files
2026-04-28T20:08:44.776460287Z INFO: Waiting for node performance related collection to complete ...
2026-04-28T20:08:44.887417133Z INFO: Collecting Insights Archives from insights-operator-585dfdc468-xq7sp
2026-04-28T20:08:44.887473506Z insights-runtime-extractor-6t8k2
2026-04-28T20:08:44.887495584Z insights-runtime-extractor-b7hh9
2026-04-28T20:08:44.887532611Z insights-runtime-extractor-gtt2h
2026-04-28T20:08:44.966341882Z No resources found in openshift-etcd namespace.
2026-04-28T20:08:45.188113808Z INFO: "metallb-operator" not detected. Skipping.
2026-04-28T20:08:45.287639147Z No resources found
2026-04-28T20:08:45.327638991Z INFO: "sriov-network-operator" not detected. Skipping.
2026-04-28T20:08:45.494789001Z INFO: Found 1 replicas - prometheus-k8s-0
2026-04-28T20:08:45.531757458Z error: the server doesn't have a resource type "performanceprofile"
2026-04-28T20:08:45.715808676Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-04-28T20:08:45.734281752Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-04-28T20:08:45.734492225Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-04-28T20:08:45.734492225Z See 'oc rsync -h' for help and examples
2026-04-28T20:08:45.742905921Z Wrote inspect data to must-gather.
2026-04-28T20:08:45.757841340Z INFO: Worker host service log collection to complete.
2026-04-28T20:08:45.829991390Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-04-28T20:08:45.830072841Z INFO: on-disk MachineConfig config collection complete.
2026-04-28T20:08:45.981046413Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-04-28T20:08:46.122268819Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-28T20:08:46.288638834Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-04-28T20:08:46.415717572Z INFO: 'previous-logs' folder not found on ip-10-0-138-34.ec2.internal, skipping...
2026-04-28T20:08:46.428353880Z INFO: Waiting for HAProxy config collection to complete ...
2026-04-28T20:08:46.571719606Z Wrote inspect data to must-gather.
2026-04-28T20:08:46.642205382Z No resources found
2026-04-28T20:08:46.666187309Z Gathering data for ns/openshift-monitoring...
2026-04-28T20:08:46.670089020Z Wrote inspect data to must-gather.
2026-04-28T20:08:46.727253575Z Wrote inspect data to must-gather.
2026-04-28T20:08:46.765199164Z Wrote inspect data to must-gather.
2026-04-28T20:08:46.786894394Z ERROR: No running kube-apiserver pods found
2026-04-28T20:08:46.796289797Z Wrote inspect data to must-gather.
2026-04-28T20:08:46.833405259Z Wrote inspect data to must-gather.
2026-04-28T20:08:46.936209608Z tar: Removing leading `/' from member names
2026-04-28T20:08:46.936441492Z error: the server doesn't have a resource type "machineconfigs"
2026-04-28T20:08:46.945384644Z INFO: HAProxy config collection complete.
2026-04-28T20:08:46.983095222Z INFO: 'previous-logs' folder not found on ip-10-0-139-184.ec2.internal, skipping...
2026-04-28T20:08:47.031134647Z INFO: Getting rules from prometheus-k8s-0
2026-04-28T20:08:47.075200949Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-28T20:08:47.120727389Z Wrote inspect data to must-gather.
2026-04-28T20:08:47.188127826Z Wrote inspect data to must-gather.
2026-04-28T20:08:47.255474354Z INFO: 'previous-logs' folder not found on ip-10-0-140-230.ec2.internal, skipping...
2026-04-28T20:08:47.255511893Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-04-28T20:08:47.255511893Z INFO: Machine Config Daemon termination log collection complete.
2026-04-28T20:08:47.359921391Z Wrote inspect data to must-gather.
2026-04-28T20:08:47.408622492Z Wrote inspect data to must-gather.
2026-04-28T20:08:47.500684557Z INFO: Getting status/config from prometheus-k8s-0
2026-04-28T20:08:47.573406676Z error: the server doesn't have a resource type "kubeletconfigs"
2026-04-28T20:08:47.601908000Z Wrote inspect data to must-gather.
2026-04-28T20:08:47.776984072Z Wrote inspect data to must-gather.
2026-04-28T20:08:47.896550343Z INFO: INTERCONNECT MODE
2026-04-28T20:08:47.896581272Z INFO: Gathering ovn-kubernetes DBs
2026-04-28T20:08:47.931051170Z INFO: Getting status/flags from prometheus-k8s-0
2026-04-28T20:08:47.992276552Z Wrote inspect data to must-gather.
2026-04-28T20:08:48.062621998Z INFO: Gathering OVN_Northbound from ovnkube-node-5tdk8...
2026-04-28T20:08:48.062889545Z INFO: Gathering OVN_Northbound from ovnkube-node-7xms2...
2026-04-28T20:08:48.063124325Z INFO: Gathering OVN_Northbound from ovnkube-node-wv7qh...
2026-04-28T20:08:48.063397768Z INFO: Gathering OVN_Southbound from ovnkube-node-5tdk8...
2026-04-28T20:08:48.063653487Z INFO: Gathering OVN_Southbound from ovnkube-node-7xms2...
2026-04-28T20:08:48.063936594Z INFO: Gathering OVN_Southbound from ovnkube-node-wv7qh...
2026-04-28T20:08:48.184307891Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-04-28T20:08:48.313231854Z Wrote inspect data to must-gather.
2026-04-28T20:08:48.323445613Z tar: Removing leading `/' from member names
2026-04-28T20:08:48.354933863Z tar: Removing leading `/' from member names
2026-04-28T20:08:48.355574530Z tar: Removing leading `/' from member names
2026-04-28T20:08:48.363162363Z tar: Removing leading `/' from member names
2026-04-28T20:08:48.373454394Z tar: Removing leading `/' from member names
2026-04-28T20:08:48.373741343Z tar: Removing leading `/' from member names
2026-04-28T20:08:48.496916458Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-04-28T20:08:48.549845429Z Wrote inspect data to must-gather.
2026-04-28T20:08:48.686407653Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-04-28T20:08:48.802407874Z Wrote inspect data to must-gather.
2026-04-28T20:08:48.803713372Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-04-28T20:08:48.923943597Z INFO: Waiting for network log collection to complete ...
2026-04-28T20:08:48.924007209Z INFO: Waiting for ovnk database copies to complete ...
2026-04-28T20:08:48.924638174Z INFO: Copying ovnk databases complete.
2026-04-28T20:08:48.926870263Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-04-28T20:08:48.929112311Z ovnk_database_store/
2026-04-28T20:08:48.929112311Z ovnk_database_store/ovnkube-node-7xms2_sbdb
2026-04-28T20:08:48.954734256Z ovnk_database_store/ovnkube-node-wv7qh_nbdb
2026-04-28T20:08:48.963368201Z ovnk_database_store/ovnkube-node-wv7qh_sbdb
2026-04-28T20:08:48.981463945Z ovnk_database_store/ovnkube-node-5tdk8_sbdb
2026-04-28T20:08:49.002566460Z ovnk_database_store/ovnkube-node-5tdk8_nbdb
2026-04-28T20:08:49.014350079Z ovnk_database_store/ovnkube-node-7xms2_nbdb
2026-04-28T20:08:49.027524193Z INFO: Network log collection complete.
2026-04-28T20:08:49.035403118Z Wrote inspect data to must-gather.
2026-04-28T20:08:49.217974248Z INFO: Getting status from alertmanager-main-0
2026-04-28T20:08:49.265440721Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-04-28T20:08:49.333887634Z Wrote inspect data to must-gather.
2026-04-28T20:08:49.443571476Z Gathering data for ns/openshift-network-console...
2026-04-28T20:08:49.602023095Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-04-28T20:08:49.604053891Z INFO: Fallback to identify the container image from release info
2026-04-28T20:08:49.854588248Z Gathering data for ns/openshift-console-operator...
2026-04-28T20:08:50.192546417Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f2123db9953f98da0e43c266e6a8a070bf221533b995f87b7e358cc7498ca6d
2026-04-28T20:08:50.222575000Z Gathering data for ns/openshift-console...
2026-04-28T20:08:50.312889413Z daemonset.apps/perf-node-gather-daemonset created
2026-04-28T20:08:50.451917012Z Waiting for performance profile collector pods to become ready: 1
2026-04-28T20:08:50.695162888Z Gathering data for ns/openshift-cluster-storage-operator...
2026-04-28T20:08:51.066773895Z Gathering data for ns/openshift-dns-operator...
2026-04-28T20:08:51.359342244Z Gathering data for ns/openshift-dns...
2026-04-28T20:08:51.573521895Z Waiting for performance profile collector pods to become ready: 2
2026-04-28T20:08:52.080539645Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-28T20:08:52.101198289Z Gathering data for ns/openshift-image-registry...
2026-04-28T20:08:52.607800623Z Gathering data for ns/openshift-ingress-operator...
2026-04-28T20:08:52.708439287Z Waiting for performance profile collector pods to become ready: 3
2026-04-28T20:08:52.976267141Z Gathering data for ns/openshift-ingress...
2026-04-28T20:08:53.319708059Z Gathering data for ns/openshift-ingress-canary...
2026-04-28T20:08:53.706792129Z Gathering data for ns/openshift-insights...
2026-04-28T20:08:53.825669753Z Waiting for performance profile collector pods to become ready: 4
2026-04-28T20:08:54.941349722Z Waiting for performance profile collector pods to become ready: 5
2026-04-28T20:08:55.902813134Z Gathering data for ns/kserve...
2026-04-28T20:08:56.073681561Z Waiting for performance profile collector pods to become ready: 6
2026-04-28T20:08:57.085021662Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-28T20:08:57.214728941Z Waiting for performance profile collector pods to become ready: 7
2026-04-28T20:08:57.773499078Z Gathering data for ns/openshift-config...
2026-04-28T20:08:58.102547351Z Gathering data for ns/openshift-config-managed...
2026-04-28T20:08:58.348487343Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-04-28T20:08:58.576294059Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-04-28T20:08:58.646300244Z Collecting performance related data for node ip-10-0-140-230.ec2.internal
2026-04-28T20:08:58.654638779Z Collecting performance related data for node ip-10-0-139-184.ec2.internal
2026-04-28T20:08:58.662186263Z Collecting performance related data for node ip-10-0-138-34.ec2.internal
2026-04-28T20:08:58.942146240Z Gathering data for ns/openshift-kube-apiserver...
2026-04-28T20:08:59.328427908Z Gathering data for ns/openshift-kube-controller-manager...
2026-04-28T20:08:59.692866636Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-04-28T20:09:00.080958444Z Gathering data for ns/openshift-kube-scheduler...
2026-04-28T20:09:00.414697331Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-04-28T20:09:00.714480516Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-04-28T20:09:01.071991022Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-04-28T20:09:01.482784680Z Gathering data for ns/openshift-user-workload-monitoring...
2026-04-28T20:09:02.056823864Z Gathering data for ns/openshift-multus...
2026-04-28T20:09:02.090057706Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-28T20:09:03.607893657Z Gathering data for ns/openshift-ovn-kubernetes...
2026-04-28T20:09:05.328226311Z Gathering data for ns/openshift-host-network...
2026-04-28T20:09:05.631439427Z Gathering data for ns/openshift-network-diagnostics...
2026-04-28T20:09:06.191084585Z Gathering data for ns/openshift-network-node-identity...
2026-04-28T20:09:06.650032639Z Gathering data for ns/openshift-network-operator...
2026-04-28T20:09:07.009367401Z Collecting kubelet logs for node ip-10-0-140-230.ec2.internal
2026-04-28T20:09:07.011493036Z Collecting kubelet logs for node ip-10-0-139-184.ec2.internal
2026-04-28T20:09:07.013918760Z Collecting kubelet logs for node ip-10-0-138-34.ec2.internal
2026-04-28T20:09:07.064268693Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-04-28T20:09:07.098670314Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-28T20:09:07.409309735Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-04-28T20:09:07.419705267Z daemonset.apps "perf-node-gather-daemonset" deleted
2026-04-28T20:09:07.425972892Z INFO: Node performance data collection complete.
2026-04-28T20:09:08.029431650Z Gathering data for ns/openshift-apiserver-operator...
2026-04-28T20:09:08.372510195Z Gathering data for ns/openshift-apiserver...
2026-04-28T20:09:08.766168491Z Gathering data for ns/openshift-controller-manager-operator...
2026-04-28T20:09:09.072404967Z Gathering data for ns/openshift-controller-manager...
2026-04-28T20:09:09.344809304Z Gathering data for ns/openshift-cluster-samples-operator...
2026-04-28T20:09:10.004203989Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-04-28T20:09:10.295919799Z Gathering data for ns/openshift-service-ca-operator...
2026-04-28T20:09:10.603646328Z Gathering data for ns/openshift-service-ca...
2026-04-28T20:09:10.961161431Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-04-28T20:09:11.882957739Z Wrote inspect data to must-gather.
2026-04-28T20:09:11.883000084Z error: inspection completed with the errors occurred while gathering data:
2026-04-28T20:09:11.883000084Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-04-28T20:09:12.020783477Z error: the server doesn't have a resource type "clusters"
2026-04-28T20:09:12.104491659Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-04-28T20:09:12.175720889Z Caches written to disk