2026-07-07T14:08:42.651051621Z [disk usage checker] Started
2026-07-07T14:08:42.654958419Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-07-07T14:08:42.869112235Z Gathering data for ns/openshift-cluster-version...
2026-07-07T14:08:43.017247411Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T14:08:43.125175837Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:08:43.216870965Z Gathering data for ns/default...
2026-07-07T14:08:43.559810721Z Gathering data for ns/openshift...
2026-07-07T14:08:44.053040463Z Gathering data for ns/kube-system...
2026-07-07T14:08:44.819525948Z Gathering data for ns/openshift-etcd...
2026-07-07T14:08:45.185936996Z Wrote inspect data to must-gather.
2026-07-07T14:08:45.185972386Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T14:08:45.185972386Z     namespaces "assisted-installer" not found
2026-07-07T14:08:45.655875249Z Waiting on subprocesses to finish execution.
2026-07-07T14:08:45.661126885Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T14:08:45.673582703Z INFO: Gathering HAProxy config files
2026-07-07T14:08:45.679686661Z Executing Istio gather script
2026-07-07T14:08:45.680461693Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T14:08:45.681869934Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T14:08:45.704363692Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T14:08:45.712267456Z INFO: Collecting host service logs for crio
2026-07-07T14:08:45.712640461Z INFO: Collecting host service logs for kubelet
2026-07-07T14:08:45.712962463Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T14:08:45.713289305Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T14:08:45.713666139Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T14:08:45.713978719Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T14:08:45.714288947Z INFO: Collecting host service logs for NetworkManager
2026-07-07T14:08:45.714615148Z INFO: Collecting host service logs for openvswitch
2026-07-07T14:08:45.714908416Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T14:08:45.715216645Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T14:08:45.715547398Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T14:08:45.715909759Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T14:08:46.468143670Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T14:08:46.562713041Z Wrote inspect data to must-gather.
2026-07-07T14:08:46.617893362Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T14:08:46.747665222Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T14:08:46.862680216Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-294tt
2026-07-07T14:08:46.862734223Z insights-runtime-extractor-fcvkp
2026-07-07T14:08:46.862753279Z insights-runtime-extractor-tx5hn
2026-07-07T14:08:46.862771729Z insights-runtime-extractor-zgztt
2026-07-07T14:08:46.938784502Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T14:08:46.989094885Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T14:08:46.989175685Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T14:08:47.010947674Z No resources found in openshift-etcd namespace.
2026-07-07T14:08:47.211621443Z Wrote inspect data to must-gather.
2026-07-07T14:08:47.231151793Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T14:08:47.265078021Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T14:08:47.412115802Z INFO: Worker host service log collection to complete.
2026-07-07T14:08:47.517747325Z No resources found
2026-07-07T14:08:47.706332794Z ERROR: No running kube-apiserver pods found
2026-07-07T14:08:47.768679180Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-07-07T14:08:47.813106253Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T14:08:48.028384238Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:08:48.101112806Z Wrote inspect data to must-gather.
2026-07-07T14:08:48.161273852Z Wrote inspect data to must-gather.
2026-07-07T14:08:48.258488208Z Wrote inspect data to must-gather.
2026-07-07T14:08:48.305527286Z No resources found
2026-07-07T14:08:48.329076770Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T14:08:48.329076770Z See 'oc rsync -h' for help and examples
2026-07-07T14:08:48.332231768Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T14:08:48.430659883Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T14:08:48.443598488Z Wrote inspect data to must-gather.
2026-07-07T14:08:48.475863205Z Gathering data for ns/openshift-monitoring...
2026-07-07T14:08:48.527635123Z Wrote inspect data to must-gather.
2026-07-07T14:08:48.582574080Z INFO: 'previous-logs' folder not found on ip-10-0-129-77.ec2.internal, skipping...
2026-07-07T14:08:48.635618748Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T14:08:48.742033001Z Wrote inspect data to must-gather.
2026-07-07T14:08:48.770728776Z tar: Removing leading `/' from member names
2026-07-07T14:08:48.790638590Z INFO: HAProxy config collection complete.
2026-07-07T14:08:48.810569129Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T14:08:48.857882845Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T14:08:48.943089540Z INFO: 'previous-logs' folder not found on ip-10-0-132-35.ec2.internal, skipping...
2026-07-07T14:08:49.094548464Z Wrote inspect data to must-gather.
2026-07-07T14:08:49.158006557Z Wrote inspect data to must-gather.
2026-07-07T14:08:49.222685958Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T14:08:49.266342328Z INFO: 'previous-logs' folder not found on ip-10-0-135-115.ec2.internal, skipping...
2026-07-07T14:08:49.266381714Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T14:08:49.266418355Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T14:08:49.304243085Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T14:08:49.357875413Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T14:08:49.385382051Z Wrote inspect data to must-gather.
2026-07-07T14:08:49.615251652Z Wrote inspect data to must-gather.
2026-07-07T14:08:49.718044967Z INFO: INTERCONNECT MODE
2026-07-07T14:08:49.718082062Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T14:08:49.765695539Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T14:08:49.853813052Z Wrote inspect data to must-gather.
2026-07-07T14:08:49.912588205Z INFO: Gathering OVN_Northbound from ovnkube-node-jdkc9...
2026-07-07T14:08:49.912803610Z INFO: Gathering OVN_Northbound from ovnkube-node-jxfcf...
2026-07-07T14:08:49.912987519Z INFO: Gathering OVN_Northbound from ovnkube-node-zrnmw...
2026-07-07T14:08:49.913278137Z INFO: Gathering OVN_Southbound from ovnkube-node-jdkc9...
2026-07-07T14:08:49.913494956Z INFO: Gathering OVN_Southbound from ovnkube-node-jxfcf...
2026-07-07T14:08:49.914061845Z INFO: Gathering OVN_Southbound from ovnkube-node-zrnmw...
2026-07-07T14:08:50.101800160Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T14:08:50.248394978Z tar: Removing leading `/' from member names
2026-07-07T14:08:50.269624641Z tar: Removing leading `/' from member names
2026-07-07T14:08:50.269624641Z Wrote inspect data to must-gather.
2026-07-07T14:08:50.275237803Z tar: Removing leading `/' from member names
2026-07-07T14:08:50.285390559Z tar: Removing leading `/' from member names
2026-07-07T14:08:50.293654022Z tar: Removing leading `/' from member names
2026-07-07T14:08:50.319806819Z tar: Removing leading `/' from member names
2026-07-07T14:08:50.380948184Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T14:08:50.424708406Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T14:08:50.452268318Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T14:08:50.569979491Z Wrote inspect data to must-gather.
2026-07-07T14:08:50.704182780Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:50.748889954Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T14:08:50.846607363Z Wrote inspect data to must-gather.
2026-07-07T14:08:50.873649029Z INFO: Waiting for network log collection to complete ...
2026-07-07T14:08:50.873690199Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T14:08:50.874127440Z INFO: Copying ovnk databases complete.
2026-07-07T14:08:50.876114045Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-07-07T14:08:50.878635619Z ovnk_database_store/
2026-07-07T14:08:50.878635619Z ovnk_database_store/ovnkube-node-jdkc9_nbdb
2026-07-07T14:08:50.889421046Z ovnk_database_store/ovnkube-node-jdkc9_sbdb
2026-07-07T14:08:50.912580761Z ovnk_database_store/ovnkube-node-zrnmw_sbdb
2026-07-07T14:08:50.932256788Z ovnk_database_store/ovnkube-node-jxfcf_sbdb
2026-07-07T14:08:50.948463759Z ovnk_database_store/ovnkube-node-jxfcf_nbdb
2026-07-07T14:08:50.955080157Z ovnk_database_store/ovnkube-node-zrnmw_nbdb
2026-07-07T14:08:50.965689832Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:50.966746209Z INFO: Network log collection complete.
2026-07-07T14:08:50.968551757Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T14:08:51.099618516Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T14:08:51.195344559Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:51.205678108Z Wrote inspect data to must-gather.
2026-07-07T14:08:51.230705200Z INFO: Getting status from alertmanager-main-0
2026-07-07T14:08:51.330879645Z Gathering data for ns/openshift-network-console...
2026-07-07T14:08:51.450531348Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:51.455711449Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T14:08:51.537686317Z Wrote inspect data to must-gather.
2026-07-07T14:08:51.664014855Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:51.841759169Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T14:08:51.844186742Z INFO: Fallback to identify the container image from release info
2026-07-07T14:08:51.867957448Z Gathering data for ns/openshift-console-operator...
2026-07-07T14:08:51.903223911Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:51.905696881Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T14:08:52.145677434Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:52.311240308Z Gathering data for ns/openshift-console...
2026-07-07T14:08:52.369187822Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:52.372266838Z Inspecting resource crd/inferencemodels.inference.networking.x-k8s.io
2026-07-07T14:08:52.553871727Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T14:08:52.568691538Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:52.699227119Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T14:08:52.797356406Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-07-07T14:08:52.838882723Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:52.841451686Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-07T14:08:52.867403161Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T14:08:52.912485104Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T14:08:53.038223108Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:53.273634016Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:53.277837262Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-07T14:08:53.474015661Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:53.536471906Z Gathering data for ns/openshift-dns-operator...
2026-07-07T14:08:53.687420115Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:53.689940060Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-07T14:08:53.878672129Z Gathering data for ns/openshift-dns...
2026-07-07T14:08:53.883372786Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:54.012912500Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T14:08:54.085035915Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:54.087731854Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T14:08:54.266875301Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:54.478048711Z Wrote inspect data to /must-gather/istio.
2026-07-07T14:08:54.591707523Z Gathering data for ns/openshift-image-registry...
2026-07-07T14:08:55.153395123Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T14:08:55.163181159Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T14:08:55.250858545Z Done executing Istio gather script
2026-07-07T14:08:55.553874254Z Gathering data for ns/openshift-ingress...
2026-07-07T14:08:55.934141361Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T14:08:56.301688869Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T14:08:56.382429300Z Gathering data for ns/openshift-insights...
2026-07-07T14:08:57.431200904Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T14:08:57.802398344Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-07-07T14:08:58.575603877Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T14:08:59.354626976Z Gathering data for ns/openshift-config...
2026-07-07T14:08:59.664181832Z Gathering data for ns/openshift-config-managed...
2026-07-07T14:08:59.721716609Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T14:09:00.153631769Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T14:09:00.458586403Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T14:09:00.863524928Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T14:09:00.866222386Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T14:09:01.150786775Z Collecting performance related data for node ip-10-0-135-115.ec2.internal
2026-07-07T14:09:01.163902390Z Collecting performance related data for node ip-10-0-132-35.ec2.internal
2026-07-07T14:09:01.170067380Z Collecting performance related data for node ip-10-0-129-77.ec2.internal
2026-07-07T14:09:01.265748885Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T14:09:01.699797750Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T14:09:02.138300523Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T14:09:02.601766056Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T14:09:02.807603460Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-07-07T14:09:03.021747263Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T14:09:03.369377385Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T14:09:03.980264473Z Gathering data for ns/openshift-multus...
2026-07-07T14:09:05.670492747Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T14:09:07.187938382Z Gathering data for ns/openshift-host-network...
2026-07-07T14:09:07.622838363Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T14:09:07.814857642Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-07-07T14:09:08.244395268Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T14:09:08.743307710Z Gathering data for ns/openshift-network-operator...
2026-07-07T14:09:09.200977066Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T14:09:09.565245809Z Collecting kubelet logs for node ip-10-0-135-115.ec2.internal
2026-07-07T14:09:09.567257889Z Collecting kubelet logs for node ip-10-0-132-35.ec2.internal
2026-07-07T14:09:09.569472950Z Collecting kubelet logs for node ip-10-0-129-77.ec2.internal
2026-07-07T14:09:09.618770839Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T14:09:09.965485984Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-nbwhl namespace
2026-07-07T14:09:09.970951762Z INFO: Node performance data collection complete.
2026-07-07T14:09:10.339607618Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T14:09:10.734624820Z Gathering data for ns/openshift-apiserver...
2026-07-07T14:09:11.189741918Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T14:09:11.564420992Z Gathering data for ns/openshift-controller-manager...
2026-07-07T14:09:11.920333479Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T14:09:12.633486176Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T14:09:12.819888733Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-07-07T14:09:13.013350077Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T14:09:13.372289772Z Gathering data for ns/openshift-service-ca...
2026-07-07T14:09:13.824540668Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T14:09:14.809887329Z Wrote inspect data to must-gather.
2026-07-07T14:09:14.809931335Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T14:09:14.809931335Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T14:09:14.955957452Z error: the server doesn't have a resource type "clusters"
2026-07-07T14:09:15.085090198Z Caches written to disk