2026-07-06T15:38:37.378612392Z [disk usage checker] Started
2026-07-06T15:38:37.381909223Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T15:38:37.582800713Z Gathering data for ns/openshift-cluster-version...
2026-07-06T15:38:37.735340741Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-06T15:38:37.834963958Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T15:38:37.976308294Z Gathering data for ns/default...
2026-07-06T15:38:38.327499065Z Gathering data for ns/openshift...
2026-07-06T15:38:38.809694247Z Gathering data for ns/kube-system...
2026-07-06T15:38:39.623300804Z Gathering data for ns/openshift-etcd...
2026-07-06T15:38:39.985915991Z Wrote inspect data to must-gather.
2026-07-06T15:38:39.986001557Z error: inspection completed with the errors occurred while gathering data:
2026-07-06T15:38:39.986001557Z     namespaces "assisted-installer" not found
2026-07-06T15:38:40.305535650Z Waiting on subprocesses to finish execution.
2026-07-06T15:38:40.311838007Z INFO: Collecting host service logs for crio
2026-07-06T15:38:40.312152293Z INFO: Collecting host service logs for kubelet
2026-07-06T15:38:40.312488888Z INFO: Collecting host service logs for rpm-ostreed
2026-07-06T15:38:40.312903759Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-06T15:38:40.313249408Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-06T15:38:40.313572099Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-06T15:38:40.313933500Z INFO: Collecting host service logs for NetworkManager
2026-07-06T15:38:40.314255308Z INFO: Collecting host service logs for openvswitch
2026-07-06T15:38:40.314580361Z INFO: Collecting host service logs for ovs-configuration
2026-07-06T15:38:40.314957317Z INFO: Collecting host service logs for ovsdb-server
2026-07-06T15:38:40.315306621Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-06T15:38:40.315703591Z INFO: Waiting for worker host service log collection to complete ...
2026-07-06T15:38:40.316029477Z INFO: Gathering HAProxy config files
2026-07-06T15:38:40.318068584Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-06T15:38:40.318387726Z Executing Istio gather script
2026-07-06T15:38:40.327844476Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-06T15:38:40.344880248Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-06T15:38:40.404988340Z INFO: Waiting for node performance related collection to complete ...
2026-07-06T15:38:41.313002792Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-06T15:38:41.327647716Z INFO: "metallb-operator" not detected. Skipping.
2026-07-06T15:38:41.472317854Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-06T15:38:41.637753076Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-06T15:38:41.784910762Z No resources found in openshift-etcd namespace.
2026-07-06T15:38:41.859499326Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-06T15:38:41.882493949Z INFO: Worker host service log collection to complete.
2026-07-06T15:38:41.935896266Z error: the server doesn't have a resource type "performanceprofile"
2026-07-06T15:38:42.016612527Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-jzrlc
2026-07-06T15:38:42.016612527Z insights-runtime-extractor-jrlbv
2026-07-06T15:38:42.016612527Z insights-runtime-extractor-ppxpt
2026-07-06T15:38:42.016612527Z insights-runtime-extractor-s4fdk
2026-07-06T15:38:42.134665973Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T15:38:42.151238746Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-06T15:38:42.172393805Z Wrote inspect data to must-gather.
2026-07-06T15:38:42.196013439Z Wrote inspect data to must-gather.
2026-07-06T15:38:42.261414178Z No resources found
2026-07-06T15:38:42.410259363Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-06T15:38:42.410357318Z INFO: on-disk MachineConfig config collection complete.
2026-07-06T15:38:42.428962865Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T15:38:42.557842458Z ERROR: No running kube-apiserver pods found
2026-07-06T15:38:42.578012193Z Wrote inspect data to must-gather.
2026-07-06T15:38:42.586317508Z No resources found
2026-07-06T15:38:42.598141678Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-06T15:38:42.632948383Z tar: Removing leading `/' from member names
2026-07-06T15:38:42.647622071Z INFO: HAProxy config collection complete.
2026-07-06T15:38:42.758183532Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-06T15:38:42.758183532Z See 'oc rsync -h' for help and examples
2026-07-06T15:38:42.782196149Z Wrote inspect data to must-gather.
2026-07-06T15:38:42.867239768Z INFO: 'previous-logs' folder not found on ip-10-0-134-165.ec2.internal, skipping...
2026-07-06T15:38:42.868349009Z Wrote inspect data to must-gather.
2026-07-06T15:38:42.901901043Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-06T15:38:42.966761599Z Wrote inspect data to must-gather.
2026-07-06T15:38:43.002876558Z Wrote inspect data to must-gather.
2026-07-06T15:38:43.084205680Z Wrote inspect data to must-gather.
2026-07-06T15:38:43.171613942Z Gathering data for ns/openshift-monitoring...
2026-07-06T15:38:43.201368954Z error: the server doesn't have a resource type "machineconfigs"
2026-07-06T15:38:43.242955800Z INFO: 'previous-logs' folder not found on ip-10-0-134-23.ec2.internal, skipping...
2026-07-06T15:38:43.399441427Z Wrote inspect data to must-gather.
2026-07-06T15:38:43.403799808Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-06T15:38:43.437671661Z INFO: Getting rules from prometheus-k8s-0
2026-07-06T15:38:43.587376582Z INFO: 'previous-logs' folder not found on ip-10-0-140-57.ec2.internal, skipping...
2026-07-06T15:38:43.587513156Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-06T15:38:43.587572990Z INFO: Machine Config Daemon termination log collection complete.
2026-07-06T15:38:43.686213558Z Wrote inspect data to must-gather.
2026-07-06T15:38:43.704914321Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-06T15:38:43.706574269Z Wrote inspect data to must-gather.
2026-07-06T15:38:43.865945855Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-06T15:38:43.902696160Z INFO: Getting status/config from prometheus-k8s-0
2026-07-06T15:38:44.023894076Z INFO: INTERCONNECT MODE
2026-07-06T15:38:44.023928386Z INFO: Gathering ovn-kubernetes DBs
2026-07-06T15:38:44.104253246Z Wrote inspect data to must-gather.
2026-07-06T15:38:44.189385212Z INFO: Gathering OVN_Northbound from ovnkube-node-99jk8...
2026-07-06T15:38:44.189696568Z INFO: Gathering OVN_Northbound from ovnkube-node-9cg79...
2026-07-06T15:38:44.190037282Z INFO: Gathering OVN_Northbound from ovnkube-node-whv5v...
2026-07-06T15:38:44.190378058Z INFO: Gathering OVN_Southbound from ovnkube-node-99jk8...
2026-07-06T15:38:44.190636894Z INFO: Gathering OVN_Southbound from ovnkube-node-9cg79...
2026-07-06T15:38:44.190927668Z INFO: Gathering OVN_Southbound from ovnkube-node-whv5v...
2026-07-06T15:38:44.493084639Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-06T15:38:44.493084639Z Wrote inspect data to must-gather.
2026-07-06T15:38:44.531932841Z tar: Removing leading `/' from member names
2026-07-06T15:38:44.540967637Z tar: Removing leading `/' from member names
2026-07-06T15:38:44.565615466Z tar: Removing leading `/' from member names
2026-07-06T15:38:44.576274769Z tar: Removing leading `/' from member names
2026-07-06T15:38:44.593949506Z tar: Removing leading `/' from member names
2026-07-06T15:38:44.596185687Z tar: Removing leading `/' from member names
2026-07-06T15:38:44.748696623Z Wrote inspect data to must-gather.
2026-07-06T15:38:44.777334192Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-06T15:38:44.875214213Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-06T15:38:45.021054736Z Wrote inspect data to must-gather.
2026-07-06T15:38:45.065951761Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-06T15:38:45.088801891Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:45.094027913Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-06T15:38:45.124372708Z INFO: Waiting for network log collection to complete ...
2026-07-06T15:38:45.124417602Z INFO: Waiting for ovnk database copies to complete ...
2026-07-06T15:38:45.124935075Z INFO: Copying ovnk databases complete.
2026-07-06T15:38:45.126546406Z 3.1M	must-gather/network_logs/ovnk_database_store
2026-07-06T15:38:45.128705016Z ovnk_database_store/
2026-07-06T15:38:45.128705016Z ovnk_database_store/ovnkube-node-9cg79_sbdb
2026-07-06T15:38:45.147741046Z ovnk_database_store/ovnkube-node-99jk8_nbdb
2026-07-06T15:38:45.157329862Z ovnk_database_store/ovnkube-node-9cg79_nbdb
2026-07-06T15:38:45.167474453Z ovnk_database_store/ovnkube-node-whv5v_sbdb
2026-07-06T15:38:45.189576162Z ovnk_database_store/ovnkube-node-99jk8_sbdb
2026-07-06T15:38:45.210987998Z ovnk_database_store/ovnkube-node-whv5v_nbdb
2026-07-06T15:38:45.223077224Z INFO: Network log collection complete.
2026-07-06T15:38:45.286535834Z Wrote inspect data to must-gather.
2026-07-06T15:38:45.339628489Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-06T15:38:45.344317832Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:45.347898543Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-06T15:38:45.516413855Z Wrote inspect data to must-gather.
2026-07-06T15:38:45.578356605Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:45.709609286Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T15:38:45.780478737Z INFO: Getting status from alertmanager-main-0
2026-07-06T15:38:45.822019754Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:45.824630956Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-06T15:38:45.837335322Z Wrote inspect data to must-gather.
2026-07-06T15:38:45.932592738Z Gathering data for ns/openshift-network-console...
2026-07-06T15:38:46.051545589Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:46.128685309Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-06T15:38:46.131641483Z INFO: Fallback to identify the container image from release info
2026-07-06T15:38:46.270991947Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:46.275825859Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-06T15:38:46.422338751Z Gathering data for ns/openshift-console-operator...
2026-07-06T15:38:46.514306248Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:46.720207130Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:46.722983295Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-06T15:38:46.774146827Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-06T15:38:46.895957036Z Gathering data for ns/openshift-console...
2026-07-06T15:38:46.917390550Z daemonset.apps/perf-node-gather-daemonset created
2026-07-06T15:38:46.923804063Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:47.073618640Z Waiting for performance profile collector pods to become ready: 1
2026-07-06T15:38:47.151551973Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:47.153996739Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-06T15:38:47.334531817Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:47.588314742Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T15:38:47.594416492Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:47.597406282Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-06T15:38:47.654116118Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-06T15:38:47.789285118Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:47.992888730Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:47.995371945Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-06T15:38:48.187551429Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:48.207030733Z Gathering data for ns/openshift-dns-operator...
2026-07-06T15:38:48.232457905Z Waiting for performance profile collector pods to become ready: 2
2026-07-06T15:38:48.396875740Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:48.399714230Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-06T15:38:48.546085659Z Gathering data for ns/openshift-dns...
2026-07-06T15:38:48.592680906Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:48.774072949Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:48.776551863Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-06T15:38:48.946131751Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:49.143388404Z Wrote inspect data to /must-gather/istio.
2026-07-06T15:38:49.327441927Z Gathering data for ns/openshift-image-registry...
2026-07-06T15:38:49.385918537Z Waiting for performance profile collector pods to become ready: 3
2026-07-06T15:38:49.870371945Z Done executing Istio gather script
2026-07-06T15:38:49.892376993Z Gathering data for ns/openshift-ingress-operator...
2026-07-06T15:38:50.231431608Z Gathering data for ns/openshift-ingress...
2026-07-06T15:38:50.501098168Z Waiting for performance profile collector pods to become ready: 4
2026-07-06T15:38:50.620957584Z Gathering data for ns/openshift-ingress-canary...
2026-07-06T15:38:51.125711811Z Gathering data for ns/openshift-insights...
2026-07-06T15:38:51.627531900Z Waiting for performance profile collector pods to become ready: 5
2026-07-06T15:38:52.593068048Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T15:38:52.749992047Z Waiting for performance profile collector pods to become ready: 6
2026-07-06T15:38:53.391700465Z Gathering data for ns/kserve...
2026-07-06T15:38:53.891837542Z Waiting for performance profile collector pods to become ready: 7
2026-07-06T15:38:55.014983060Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-06T15:38:55.227674242Z Gathering data for ns/openshift-config...
2026-07-06T15:38:55.281863300Z Collecting performance related data for node ip-10-0-134-165.ec2.internal
2026-07-06T15:38:55.294398073Z Collecting performance related data for node ip-10-0-140-57.ec2.internal
2026-07-06T15:38:55.300985047Z Collecting performance related data for node ip-10-0-134-23.ec2.internal
2026-07-06T15:38:55.600503237Z Gathering data for ns/openshift-config-managed...
2026-07-06T15:38:56.145184275Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-06T15:38:56.513745344Z Gathering data for ns/openshift-kube-apiserver...
2026-07-06T15:38:56.882383053Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-06T15:38:57.205760489Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-06T15:38:57.559828767Z Gathering data for ns/openshift-kube-scheduler...
2026-07-06T15:38:57.602045053Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T15:38:57.890524389Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-06T15:38:58.248997807Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-06T15:38:58.643279814Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-06T15:38:58.957662271Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-06T15:38:59.506129146Z Gathering data for ns/openshift-multus...
2026-07-06T15:39:01.154249116Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-06T15:39:02.557534168Z Gathering data for ns/openshift-host-network...
2026-07-06T15:39:02.609602621Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T15:39:02.925322752Z Gathering data for ns/openshift-network-diagnostics...
2026-07-06T15:39:03.577303665Z Gathering data for ns/openshift-network-node-identity...
2026-07-06T15:39:03.790123440Z Collecting kubelet logs for node ip-10-0-134-165.ec2.internal
2026-07-06T15:39:03.792617458Z Collecting kubelet logs for node ip-10-0-140-57.ec2.internal
2026-07-06T15:39:03.794859051Z Collecting kubelet logs for node ip-10-0-134-23.ec2.internal
2026-07-06T15:39:04.076201877Z Gathering data for ns/openshift-network-operator...
2026-07-06T15:39:04.177170202Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-4g7kz namespace
2026-07-06T15:39:04.186300904Z INFO: Node performance data collection complete.
2026-07-06T15:39:04.522292227Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-06T15:39:04.833303606Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-06T15:39:05.313975319Z Gathering data for ns/openshift-apiserver-operator...
2026-07-06T15:39:05.665246022Z Gathering data for ns/openshift-apiserver...
2026-07-06T15:39:06.069037426Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-06T15:39:06.387451783Z Gathering data for ns/openshift-controller-manager...
2026-07-06T15:39:06.727364935Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-06T15:39:07.468957662Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-06T15:39:07.616368067Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-06T15:39:07.764204499Z Gathering data for ns/openshift-service-ca-operator...
2026-07-06T15:39:08.091237574Z Gathering data for ns/openshift-service-ca...
2026-07-06T15:39:08.522317508Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-06T15:39:09.600375642Z Wrote inspect data to must-gather.
2026-07-06T15:39:09.600418688Z error: inspection completed with the errors occurred while gathering data:
2026-07-06T15:39:09.600418688Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-06T15:39:09.737816361Z error: the server doesn't have a resource type "clusters"
2026-07-06T15:39:09.882799358Z Caches written to disk