2026-07-06T21:01:16.233271250Z [disk usage checker] Started
2026-07-06T21:01:16.236770056Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-06T21:01:16.434775081Z Gathering data for ns/openshift-cluster-version...
2026-07-06T21:01:16.588674454Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-06T21:01:16.695804771Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T21:01:16.781285319Z Gathering data for ns/default...
2026-07-06T21:01:17.145638770Z Gathering data for ns/openshift...
2026-07-06T21:01:17.623333796Z Gathering data for ns/kube-system...
2026-07-06T21:01:18.436898451Z Gathering data for ns/openshift-etcd...
2026-07-06T21:01:18.810383219Z Wrote inspect data to must-gather.
2026-07-06T21:01:18.810418553Z error: inspection completed with the errors occurred while gathering data:
2026-07-06T21:01:18.810418553Z     namespaces "assisted-installer" not found
2026-07-06T21:01:19.204262979Z Waiting on subprocesses to finish execution.
2026-07-06T21:01:19.210387713Z INFO: Gathering HAProxy config files
2026-07-06T21:01:19.212471503Z INFO: Collecting host service logs for crio
2026-07-06T21:01:19.212798547Z INFO: Collecting host service logs for kubelet
2026-07-06T21:01:19.213148669Z INFO: Collecting host service logs for rpm-ostreed
2026-07-06T21:01:19.213478160Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-06T21:01:19.213575547Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-06T21:01:19.213824603Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-06T21:01:19.214203085Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-06T21:01:19.214496105Z INFO: Collecting host service logs for NetworkManager
2026-07-06T21:01:19.214779176Z INFO: Collecting host service logs for openvswitch
2026-07-06T21:01:19.215060209Z INFO: Collecting host service logs for ovs-configuration
2026-07-06T21:01:19.215373957Z INFO: Collecting host service logs for ovsdb-server
2026-07-06T21:01:19.215689905Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-06T21:01:19.215919582Z Executing Istio gather script
2026-07-06T21:01:19.216052943Z INFO: Waiting for worker host service log collection to complete ...
2026-07-06T21:01:19.224298778Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-06T21:01:19.228860686Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-06T21:01:19.281534159Z INFO: Waiting for node performance related collection to complete ...
2026-07-06T21:01:19.793873250Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-06T21:01:20.123057239Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-06T21:01:20.240365571Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-06T21:01:20.240407871Z INFO: on-disk MachineConfig config collection complete.
2026-07-06T21:01:20.342912880Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-06T21:01:20.460515963Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-06T21:01:20.574138318Z No resources found
2026-07-06T21:01:20.587370338Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-ggr6h
2026-07-06T21:01:20.587416745Z insights-runtime-extractor-25b6p
2026-07-06T21:01:20.587438555Z insights-runtime-extractor-cmhwr
2026-07-06T21:01:20.587458648Z insights-runtime-extractor-jxl99
2026-07-06T21:01:20.677915976Z No resources found in openshift-etcd namespace.
2026-07-06T21:01:20.742919361Z INFO: Worker host service log collection to complete.
2026-07-06T21:01:20.757547457Z INFO: "metallb-operator" not detected. Skipping.
2026-07-06T21:01:20.990236836Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-06T21:01:21.172315968Z error: the server doesn't have a resource type "performanceprofile"
2026-07-06T21:01:21.210397794Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T21:01:21.218105069Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-06T21:01:21.218105069Z See 'oc rsync -h' for help and examples
2026-07-06T21:01:21.267512399Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-06T21:01:21.578269286Z No resources found
2026-07-06T21:01:21.601385255Z Wrote inspect data to must-gather.
2026-07-06T21:01:21.683498980Z Wrote inspect data to must-gather.
2026-07-06T21:01:21.709326200Z Gathering data for ns/openshift-monitoring...
2026-07-06T21:01:21.723451617Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-06T21:01:21.794269919Z tar: Removing leading `/' from member names
2026-07-06T21:01:21.799742648Z INFO: HAProxy config collection complete.
2026-07-06T21:01:21.802344884Z Wrote inspect data to must-gather.
2026-07-06T21:01:21.812254033Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-06T21:01:21.873402265Z Wrote inspect data to must-gather.
2026-07-06T21:01:21.876907566Z Wrote inspect data to must-gather.
2026-07-06T21:01:21.922604308Z ERROR: No running kube-apiserver pods found
2026-07-06T21:01:21.953313532Z Wrote inspect data to must-gather.
2026-07-06T21:01:22.048644621Z INFO: 'previous-logs' folder not found on ip-10-0-130-45.ec2.internal, skipping...
2026-07-06T21:01:22.079540459Z Wrote inspect data to must-gather.
2026-07-06T21:01:22.271215929Z Wrote inspect data to must-gather.
2026-07-06T21:01:22.317751133Z error: the server doesn't have a resource type "machineconfigs"
2026-07-06T21:01:22.399428731Z INFO: 'previous-logs' folder not found on ip-10-0-140-62.ec2.internal, skipping...
2026-07-06T21:01:22.422704199Z INFO: Getting rules from prometheus-k8s-0
2026-07-06T21:01:22.517036423Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-06T21:01:22.593572218Z Wrote inspect data to must-gather.
2026-07-06T21:01:22.727460361Z INFO: 'previous-logs' folder not found on ip-10-0-142-58.ec2.internal, skipping...
2026-07-06T21:01:22.727492711Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-06T21:01:22.727511016Z INFO: Machine Config Daemon termination log collection complete.
2026-07-06T21:01:22.806915454Z Wrote inspect data to must-gather.
2026-07-06T21:01:22.844346940Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-06T21:01:22.887676109Z Wrote inspect data to must-gather.
2026-07-06T21:01:22.909269855Z INFO: Getting status/config from prometheus-k8s-0
2026-07-06T21:01:22.990927084Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-06T21:01:23.215233853Z INFO: INTERCONNECT MODE
2026-07-06T21:01:23.215233853Z INFO: Gathering ovn-kubernetes DBs
2026-07-06T21:01:23.217362314Z Wrote inspect data to must-gather.
2026-07-06T21:01:23.403972802Z INFO: Gathering OVN_Northbound from ovnkube-node-6jbt7...
2026-07-06T21:01:23.404252100Z INFO: Gathering OVN_Northbound from ovnkube-node-gl8x4...
2026-07-06T21:01:23.404528364Z INFO: Gathering OVN_Northbound from ovnkube-node-q4vhp...
2026-07-06T21:01:23.404862332Z INFO: Gathering OVN_Southbound from ovnkube-node-6jbt7...
2026-07-06T21:01:23.405227836Z INFO: Gathering OVN_Southbound from ovnkube-node-gl8x4...
2026-07-06T21:01:23.405513234Z INFO: Gathering OVN_Southbound from ovnkube-node-q4vhp...
2026-07-06T21:01:23.424590064Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-06T21:01:23.513434994Z Wrote inspect data to must-gather.
2026-07-06T21:01:23.710057404Z tar: Removing leading `/' from member names
2026-07-06T21:01:23.724999791Z tar: Removing leading `/' from member names
2026-07-06T21:01:23.756563668Z tar: Removing leading `/' from member names
2026-07-06T21:01:23.779628330Z tar: Removing leading `/' from member names
2026-07-06T21:01:23.819863311Z tar: Removing leading `/' from member names
2026-07-06T21:01:23.822801259Z tar: Removing leading `/' from member names
2026-07-06T21:01:23.864508238Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-06T21:01:23.879969997Z Wrote inspect data to must-gather.
2026-07-06T21:01:23.961255654Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-06T21:01:24.139314997Z Wrote inspect data to must-gather.
2026-07-06T21:01:24.152383973Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-06T21:01:24.233796963Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-06T21:01:24.340513293Z INFO: Waiting for network log collection to complete ...
2026-07-06T21:01:24.340631426Z INFO: Waiting for ovnk database copies to complete ...
2026-07-06T21:01:24.341171546Z INFO: Copying ovnk databases complete.
2026-07-06T21:01:24.344878799Z 3.2M	must-gather/network_logs/ovnk_database_store
2026-07-06T21:01:24.347668971Z ovnk_database_store/
2026-07-06T21:01:24.347717028Z ovnk_database_store/ovnkube-node-gl8x4_nbdb
2026-07-06T21:01:24.357976734Z ovnk_database_store/ovnkube-node-6jbt7_nbdb
2026-07-06T21:01:24.365969652Z ovnk_database_store/ovnkube-node-q4vhp_sbdb
2026-07-06T21:01:24.385525598Z ovnk_database_store/ovnkube-node-6jbt7_sbdb
2026-07-06T21:01:24.401900380Z Wrote inspect data to must-gather.
2026-07-06T21:01:24.410135717Z ovnk_database_store/ovnkube-node-gl8x4_sbdb
2026-07-06T21:01:24.429976314Z ovnk_database_store/ovnkube-node-q4vhp_nbdb
2026-07-06T21:01:24.442415065Z INFO: Network log collection complete.
2026-07-06T21:01:24.470746707Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-06T21:01:24.480835861Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:24.653220469Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-06T21:01:24.729018393Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:24.732153806Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-06T21:01:24.744808030Z Wrote inspect data to must-gather.
2026-07-06T21:01:24.872469955Z Gathering data for ns/openshift-network-console...
2026-07-06T21:01:24.918947993Z INFO: Getting status from alertmanager-main-0
2026-07-06T21:01:24.959014605Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:25.080023878Z Wrote inspect data to must-gather.
2026-07-06T21:01:25.188221367Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:25.190742911Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-06T21:01:25.404871221Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-06T21:01:25.408431594Z INFO: Fallback to identify the container image from release info
2026-07-06T21:01:25.423820950Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:25.424176618Z Gathering data for ns/openshift-console-operator...
2026-07-06T21:01:25.649455603Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:25.652195233Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-06T21:01:25.854590855Z Gathering data for ns/openshift-console...
2026-07-06T21:01:25.877979806Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:26.079273004Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:26.081956432Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-06T21:01:26.268504979Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:26.311917526Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-06T21:01:26.376411881Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-06T21:01:26.446581056Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-06T21:01:26.491842931Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:26.494337908Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-06T21:01:26.532749471Z daemonset.apps/perf-node-gather-daemonset created
2026-07-06T21:01:26.687697904Z Waiting for performance profile collector pods to become ready: 1
2026-07-06T21:01:26.693105013Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:26.904750495Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:26.909774612Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-06T21:01:27.127983125Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:27.135213761Z Gathering data for ns/openshift-dns-operator...
2026-07-06T21:01:27.337937132Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:27.341745405Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-06T21:01:27.509109297Z Gathering data for ns/openshift-dns...
2026-07-06T21:01:27.538231379Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:27.719574039Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:27.723065879Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-06T21:01:27.822933474Z Waiting for performance profile collector pods to become ready: 2
2026-07-06T21:01:27.921650296Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:28.120992594Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:28.124428823Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-06T21:01:28.285956221Z Gathering data for ns/openshift-image-registry...
2026-07-06T21:01:28.308016351Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:28.512662024Z Wrote inspect data to /must-gather/istio.
2026-07-06T21:01:28.899624057Z Gathering data for ns/openshift-ingress-operator...
2026-07-06T21:01:28.982543136Z Waiting for performance profile collector pods to become ready: 3
2026-07-06T21:01:29.287449459Z Done executing Istio gather script
2026-07-06T21:01:29.322719986Z Gathering data for ns/openshift-ingress...
2026-07-06T21:01:29.690665737Z Gathering data for ns/openshift-ingress-canary...
2026-07-06T21:01:30.125369770Z Waiting for performance profile collector pods to become ready: 4
2026-07-06T21:01:30.150174829Z Gathering data for ns/openshift-insights...
2026-07-06T21:01:31.257986133Z Waiting for performance profile collector pods to become ready: 5
2026-07-06T21:01:31.316945846Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-06T21:01:32.388517911Z Waiting for performance profile collector pods to become ready: 6
2026-07-06T21:01:32.765892465Z Gathering data for ns/kserve...
2026-07-06T21:01:33.541760491Z Waiting for performance profile collector pods to become ready: 7
2026-07-06T21:01:34.510965015Z Gathering data for ns/openshift-config...
2026-07-06T21:01:34.672817931Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-06T21:01:34.945494245Z Collecting performance related data for node ip-10-0-140-62.ec2.internal
2026-07-06T21:01:34.947617796Z Gathering data for ns/openshift-config-managed...
2026-07-06T21:01:34.952399879Z Collecting performance related data for node ip-10-0-130-45.ec2.internal
2026-07-06T21:01:34.959945165Z Collecting performance related data for node ip-10-0-142-58.ec2.internal
2026-07-06T21:01:35.590214233Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-06T21:01:35.983243723Z Gathering data for ns/openshift-kube-apiserver...
2026-07-06T21:01:36.323886176Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-06T21:01:36.426803728Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-06T21:01:36.842641468Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-06T21:01:37.203720238Z Gathering data for ns/openshift-kube-scheduler...
2026-07-06T21:01:37.557802147Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-06T21:01:37.975650533Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-06T21:01:38.408306674Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-06T21:01:38.856954868Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-06T21:01:39.528458066Z Gathering data for ns/openshift-multus...
2026-07-06T21:01:41.329061255Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-06T21:01:41.613655660Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-06T21:01:43.196595728Z Gathering data for ns/openshift-host-network...
2026-07-06T21:01:43.449159378Z Collecting kubelet logs for node ip-10-0-140-62.ec2.internal
2026-07-06T21:01:43.451294216Z Collecting kubelet logs for node ip-10-0-130-45.ec2.internal
2026-07-06T21:01:43.453658237Z Collecting kubelet logs for node ip-10-0-142-58.ec2.internal
2026-07-06T21:01:43.604035579Z Gathering data for ns/openshift-network-diagnostics...
2026-07-06T21:01:43.842517501Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-rxdn5 namespace
2026-07-06T21:01:43.847945908Z INFO: Node performance data collection complete.
2026-07-06T21:01:44.251943931Z Gathering data for ns/openshift-network-node-identity...
2026-07-06T21:01:44.723709509Z Gathering data for ns/openshift-network-operator...
2026-07-06T21:01:45.227820746Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-06T21:01:45.569464140Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-06T21:01:46.145657489Z Gathering data for ns/openshift-apiserver-operator...
2026-07-06T21:01:46.333711369Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-06T21:01:46.517843103Z Gathering data for ns/openshift-apiserver...
2026-07-06T21:01:46.996935005Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-06T21:01:47.465363772Z Gathering data for ns/openshift-controller-manager...
2026-07-06T21:01:47.828909138Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-06T21:01:48.541667052Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-06T21:01:48.913542676Z Gathering data for ns/openshift-service-ca-operator...
2026-07-06T21:01:49.269411937Z Gathering data for ns/openshift-service-ca...
2026-07-06T21:01:49.696690291Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-06T21:01:50.809484044Z Wrote inspect data to must-gather.
2026-07-06T21:01:50.809531842Z error: inspection completed with the errors occurred while gathering data:
2026-07-06T21:01:50.809531842Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-06T21:01:50.945235618Z error: the server doesn't have a resource type "clusters"
2026-07-06T21:01:51.048583437Z Caches written to disk