2026-07-02T22:09:07.851727696Z [disk usage checker] Started
2026-07-02T22:09:07.855211890Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T22:09:08.049147317Z Gathering data for ns/openshift-cluster-version...
2026-07-02T22:09:08.183797568Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T22:09:08.270976086Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T22:09:08.354589315Z Gathering data for ns/default...
2026-07-02T22:09:08.681610063Z Gathering data for ns/openshift...
2026-07-02T22:09:09.092909504Z Gathering data for ns/kube-system...
2026-07-02T22:09:09.796996842Z Gathering data for ns/openshift-etcd...
2026-07-02T22:09:10.100247363Z Wrote inspect data to must-gather.
2026-07-02T22:09:10.100273638Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T22:09:10.100273638Z     namespaces "assisted-installer" not found
2026-07-02T22:09:10.669204838Z Waiting on subprocesses to finish execution.
2026-07-02T22:09:10.679017316Z INFO: Gathering HAProxy config files
2026-07-02T22:09:10.685914788Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T22:09:10.692503938Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T22:09:10.698615376Z INFO: Collecting host service logs for crio
2026-07-02T22:09:10.698908560Z INFO: Collecting host service logs for kubelet
2026-07-02T22:09:10.699192488Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T22:09:10.699477134Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T22:09:10.699789687Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T22:09:10.700091178Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T22:09:10.700428681Z INFO: Collecting host service logs for NetworkManager
2026-07-02T22:09:10.700746153Z INFO: Collecting host service logs for openvswitch
2026-07-02T22:09:10.701057209Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T22:09:10.701375764Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T22:09:10.701653977Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T22:09:10.701814301Z Executing Istio gather script
2026-07-02T22:09:10.702020547Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T22:09:10.732163281Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T22:09:10.925548760Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T22:09:11.229990417Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T22:09:11.449392864Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-w4x6d
2026-07-02T22:09:11.449447563Z insights-runtime-extractor-f7sz7
2026-07-02T22:09:11.449467350Z insights-runtime-extractor-kjzfh
2026-07-02T22:09:11.449486975Z insights-runtime-extractor-nfvss
2026-07-02T22:09:11.559820204Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T22:09:11.798837348Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T22:09:11.883577379Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T22:09:12.214720179Z No resources found
2026-07-02T22:09:12.218652879Z ERROR: No running kube-apiserver pods found
2026-07-02T22:09:12.219630145Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T22:09:12.228490229Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T22:09:12.228490229Z See 'oc rsync -h' for help and examples
2026-07-02T22:09:12.236413495Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T22:09:12.280003598Z No resources found in openshift-etcd namespace.
2026-07-02T22:09:12.353889134Z INFO: Worker host service log collection to complete.
2026-07-02T22:09:12.382581899Z Wrote inspect data to must-gather.
2026-07-02T22:09:12.401732457Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T22:09:12.401813038Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T22:09:12.419063286Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T22:09:12.436542187Z Wrote inspect data to must-gather.
2026-07-02T22:09:12.612139009Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T22:09:12.905070731Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T22:09:12.939189975Z INFO: 'previous-logs' folder not found on ip-10-0-130-121.ec2.internal, skipping...
2026-07-02T22:09:13.038508059Z Wrote inspect data to must-gather.
2026-07-02T22:09:13.173239705Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T22:09:13.208461338Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T22:09:13.222922597Z Wrote inspect data to must-gather.
2026-07-02T22:09:13.236005010Z No resources found
2026-07-02T22:09:13.272845048Z Wrote inspect data to must-gather.
2026-07-02T22:09:13.294519020Z Wrote inspect data to must-gather.
2026-07-02T22:09:13.457562283Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T22:09:13.515208409Z tar: Removing leading `/' from member names
2026-07-02T22:09:13.518615910Z INFO: HAProxy config collection complete.
2026-07-02T22:09:13.574878128Z Wrote inspect data to must-gather.
2026-07-02T22:09:13.582036060Z Gathering data for ns/openshift-monitoring...
2026-07-02T22:09:13.654910589Z INFO: 'previous-logs' folder not found on ip-10-0-130-230.ec2.internal, skipping...
2026-07-02T22:09:13.736916990Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T22:09:13.745872577Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T22:09:13.817252660Z Wrote inspect data to must-gather.
2026-07-02T22:09:13.994851614Z INFO: 'previous-logs' folder not found on ip-10-0-131-12.ec2.internal, skipping...
2026-07-02T22:09:13.994953823Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T22:09:13.995005869Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T22:09:14.021639943Z Wrote inspect data to must-gather.
2026-07-02T22:09:14.115009916Z Wrote inspect data to must-gather.
2026-07-02T22:09:14.115383302Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T22:09:14.199430130Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T22:09:14.286180631Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T22:09:14.376100161Z Wrote inspect data to must-gather.
2026-07-02T22:09:14.435260170Z Wrote inspect data to must-gather.
2026-07-02T22:09:14.681409373Z Wrote inspect data to must-gather.
2026-07-02T22:09:14.688582531Z INFO: INTERCONNECT MODE
2026-07-02T22:09:14.688632205Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T22:09:14.753274788Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T22:09:14.875543505Z INFO: Gathering OVN_Northbound from ovnkube-node-cd8lj...
2026-07-02T22:09:14.875804281Z INFO: Gathering OVN_Northbound from ovnkube-node-hlmv7...
2026-07-02T22:09:14.876031923Z INFO: Gathering OVN_Northbound from ovnkube-node-pfkb2...
2026-07-02T22:09:14.876335788Z INFO: Gathering OVN_Southbound from ovnkube-node-cd8lj...
2026-07-02T22:09:14.876624238Z INFO: Gathering OVN_Southbound from ovnkube-node-hlmv7...
2026-07-02T22:09:14.876912332Z INFO: Gathering OVN_Southbound from ovnkube-node-pfkb2...
2026-07-02T22:09:14.953043009Z Wrote inspect data to must-gather.
2026-07-02T22:09:15.077780328Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T22:09:15.224168455Z tar: Removing leading `/' from member names
2026-07-02T22:09:15.244382423Z tar: Removing leading `/' from member names
2026-07-02T22:09:15.251425831Z tar: Removing leading `/' from member names
2026-07-02T22:09:15.267790708Z tar: Removing leading `/' from member names
2026-07-02T22:09:15.275772883Z tar: Removing leading `/' from member names
2026-07-02T22:09:15.284815904Z tar: Removing leading `/' from member names
2026-07-02T22:09:15.331879162Z Wrote inspect data to must-gather.
2026-07-02T22:09:15.419811727Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T22:09:15.567553293Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T22:09:15.606995660Z Wrote inspect data to must-gather.
2026-07-02T22:09:15.666027987Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T22:09:15.728145749Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T22:09:15.835487658Z INFO: Waiting for network log collection to complete ...
2026-07-02T22:09:15.835526351Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T22:09:15.836010675Z INFO: Copying ovnk databases complete.
2026-07-02T22:09:15.838168791Z 4.0M	must-gather/network_logs/ovnk_database_store
2026-07-02T22:09:15.844147474Z ovnk_database_store/
2026-07-02T22:09:15.844202759Z ovnk_database_store/ovnkube-node-cd8lj_sbdb
2026-07-02T22:09:15.861476795Z Wrote inspect data to must-gather.
2026-07-02T22:09:15.876207772Z ovnk_database_store/ovnkube-node-cd8lj_nbdb
2026-07-02T22:09:15.886790220Z ovnk_database_store/ovnkube-node-hlmv7_sbdb
2026-07-02T22:09:15.895926520Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:15.914457474Z ovnk_database_store/ovnkube-node-pfkb2_nbdb
2026-07-02T22:09:15.925297317Z ovnk_database_store/ovnkube-node-pfkb2_sbdb
2026-07-02T22:09:15.951146462Z ovnk_database_store/ovnkube-node-hlmv7_nbdb
2026-07-02T22:09:15.964192016Z INFO: Network log collection complete.
2026-07-02T22:09:16.104204520Z W0702 22:09:16.104152    1508 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-02T22:09:16.109641710Z Wrote inspect data to must-gather.
2026-07-02T22:09:16.152048775Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:16.154985646Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T22:09:16.193044911Z INFO: Getting status from alertmanager-main-0
2026-07-02T22:09:16.239711887Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T22:09:16.388931446Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:16.426213062Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T22:09:16.428564613Z INFO: Fallback to identify the container image from release info
2026-07-02T22:09:16.446656147Z Gathering data for ns/openshift-network-console...
2026-07-02T22:09:16.603148345Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:16.605427224Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T22:09:16.801243451Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:16.883853286Z Gathering data for ns/openshift-console-operator...
2026-07-02T22:09:17.015051692Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:17.017612840Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T22:09:17.092323913Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-02T22:09:17.259900227Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T22:09:17.271355826Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:17.368131815Z Gathering data for ns/openshift-console...
2026-07-02T22:09:17.418830256Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T22:09:17.526156193Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:17.531255500Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-02T22:09:17.727428344Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:17.952594644Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T22:09:17.957376004Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:17.959880706Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-02T22:09:17.993938990Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T22:09:18.130450611Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:18.346507967Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:18.349307874Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-02T22:09:18.501504751Z Gathering data for ns/openshift-dns-operator...
2026-07-02T22:09:18.550339664Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:18.576178349Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T22:09:18.760235609Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:18.764459193Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-02T22:09:18.847048374Z Gathering data for ns/openshift-dns...
2026-07-02T22:09:18.938148377Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:19.120890883Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:19.123731576Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-02T22:09:19.309226060Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:19.506723823Z Gathering data for ns/openshift-image-registry...
2026-07-02T22:09:19.522028550Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:19.524638784Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T22:09:19.704881654Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:19.728597472Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T22:09:19.896657640Z Wrote inspect data to /must-gather/istio.
2026-07-02T22:09:20.050042443Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T22:09:20.413350212Z Gathering data for ns/openshift-ingress...
2026-07-02T22:09:20.621344858Z Done executing Istio gather script
2026-07-02T22:09:20.764276232Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T22:09:20.853822553Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T22:09:21.148553125Z Gathering data for ns/openshift-insights...
2026-07-02T22:09:21.968500213Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T22:09:22.999353097Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T22:09:23.092556454Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T22:09:23.479451317Z Gathering data for ns/kserve...
2026-07-02T22:09:24.227930057Z Waiting for performance profile collector pods to become ready: 7
2026-07-02T22:09:25.113459745Z Gathering data for ns/openshift-config...
2026-07-02T22:09:25.359833677Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T22:09:25.454614126Z Gathering data for ns/openshift-config-managed...
2026-07-02T22:09:25.633823531Z Collecting performance related data for node ip-10-0-131-12.ec2.internal
2026-07-02T22:09:25.640569965Z Collecting performance related data for node ip-10-0-130-121.ec2.internal
2026-07-02T22:09:25.647802729Z Collecting performance related data for node ip-10-0-130-230.ec2.internal
2026-07-02T22:09:25.940526421Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T22:09:26.280491239Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T22:09:26.637540012Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T22:09:27.013808053Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T22:09:27.372610653Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T22:09:27.654286606Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T22:09:27.984272376Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T22:09:28.004215976Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T22:09:28.377916729Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T22:09:28.769327190Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T22:09:29.293365874Z Gathering data for ns/openshift-multus...
2026-07-02T22:09:30.777610929Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T22:09:32.123690481Z Gathering data for ns/openshift-host-network...
2026-07-02T22:09:32.447905000Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T22:09:33.013804090Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T22:09:33.033437826Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T22:09:33.508225405Z Gathering data for ns/openshift-network-operator...
2026-07-02T22:09:33.743128477Z Collecting kubelet logs for node ip-10-0-131-12.ec2.internal
2026-07-02T22:09:33.746063349Z Collecting kubelet logs for node ip-10-0-130-121.ec2.internal
2026-07-02T22:09:33.748427512Z Collecting kubelet logs for node ip-10-0-130-230.ec2.internal
2026-07-02T22:09:34.080026440Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T22:09:34.164531766Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-28brx namespace
2026-07-02T22:09:34.169466550Z INFO: Node performance data collection complete.
2026-07-02T22:09:34.389445976Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T22:09:34.893547014Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T22:09:35.186352304Z Gathering data for ns/openshift-apiserver...
2026-07-02T22:09:35.576742641Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T22:09:35.876066641Z Gathering data for ns/openshift-controller-manager...
2026-07-02T22:09:36.145968289Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T22:09:36.775880500Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T22:09:37.062054881Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T22:09:37.393295776Z Gathering data for ns/openshift-service-ca...
2026-07-02T22:09:37.757335735Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T22:09:38.018571058Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T22:09:38.787952817Z Wrote inspect data to must-gather.
2026-07-02T22:09:38.787998417Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T22:09:38.787998417Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T22:09:38.916021268Z error: the server doesn't have a resource type "clusters"
2026-07-02T22:09:39.058999557Z Caches written to disk