2026-07-28T10:59:20.136770487Z [disk usage checker] Started
2026-07-28T10:59:20.139950774Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T10:59:20.352053271Z Gathering data for ns/openshift-cluster-version...
2026-07-28T10:59:20.497546041Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T10:59:20.597788826Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T10:59:20.694022756Z Gathering data for ns/default...
2026-07-28T10:59:21.053914829Z Gathering data for ns/openshift...
2026-07-28T10:59:21.587938983Z Gathering data for ns/kube-system...
2026-07-28T10:59:22.326448119Z Gathering data for ns/openshift-etcd...
2026-07-28T10:59:22.656118235Z Wrote inspect data to must-gather.
2026-07-28T10:59:22.656149006Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T10:59:22.656149006Z     namespaces "assisted-installer" not found
2026-07-28T10:59:22.875333736Z Waiting on subprocesses to finish execution.
2026-07-28T10:59:22.882984209Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T10:59:22.883118147Z INFO: Gathering HAProxy config files
2026-07-28T10:59:22.885060185Z Executing Istio gather script
2026-07-28T10:59:22.885287039Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T10:59:22.896671385Z INFO: Collecting host service logs for crio
2026-07-28T10:59:22.896933689Z INFO: Collecting host service logs for kubelet
2026-07-28T10:59:22.897191789Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T10:59:22.897444855Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T10:59:22.897706353Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T10:59:22.897983097Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T10:59:22.898236503Z INFO: Collecting host service logs for NetworkManager
2026-07-28T10:59:22.898484935Z INFO: Collecting host service logs for openvswitch
2026-07-28T10:59:22.898747845Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T10:59:22.899010829Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T10:59:22.899274580Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T10:59:22.899593151Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T10:59:22.903374127Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T10:59:22.950127525Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T10:59:23.385934349Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T10:59:23.650574216Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T10:59:23.650654647Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T10:59:23.755478400Z No resources found in openshift-etcd namespace.
2026-07-28T10:59:23.802899367Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T10:59:23.984625395Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T10:59:24.084400137Z No resources found
2026-07-28T10:59:24.265092295Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T10:59:24.277148342Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T10:59:24.314326787Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-rwz6k
2026-07-28T10:59:24.314405293Z insights-runtime-extractor-gk59v
2026-07-28T10:59:24.314426693Z insights-runtime-extractor-gkjrn
2026-07-28T10:59:24.314456078Z insights-runtime-extractor-swz2d
2026-07-28T10:59:24.663255641Z Wrote inspect data to must-gather.
2026-07-28T10:59:24.686849609Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T10:59:24.703377895Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T10:59:24.778019649Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T10:59:24.805233576Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T10:59:24.840171058Z No resources found
2026-07-28T10:59:24.963655550Z INFO: Worker host service log collection to complete.
2026-07-28T10:59:25.203656465Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T10:59:25.203656465Z See 'oc rsync -h' for help and examples
2026-07-28T10:59:25.203856472Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T10:59:25.210365657Z ERROR: No running kube-apiserver pods found
2026-07-28T10:59:25.235246156Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T10:59:25.235698336Z Wrote inspect data to must-gather.
2026-07-28T10:59:25.282553014Z Wrote inspect data to must-gather.
2026-07-28T10:59:25.297981590Z Wrote inspect data to must-gather.
2026-07-28T10:59:25.341823448Z Wrote inspect data to must-gather.
2026-07-28T10:59:25.466890347Z Gathering data for ns/openshift-monitoring...
2026-07-28T10:59:25.473086655Z Wrote inspect data to must-gather.
2026-07-28T10:59:25.493651743Z Wrote inspect data to must-gather.
2026-07-28T10:59:25.520909798Z tar: Removing leading `/' from member names
2026-07-28T10:59:25.530973644Z INFO: HAProxy config collection complete.
2026-07-28T10:59:25.532904769Z INFO: 'previous-logs' folder not found on ip-10-0-130-25.ec2.internal, skipping...
2026-07-28T10:59:25.653100431Z Wrote inspect data to must-gather.
2026-07-28T10:59:25.708186840Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T10:59:25.837526327Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T10:59:25.855031110Z INFO: 'previous-logs' folder not found on ip-10-0-131-51.ec2.internal, skipping...
2026-07-28T10:59:25.906873906Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T10:59:25.967219173Z Wrote inspect data to must-gather.
2026-07-28T10:59:26.165539055Z INFO: 'previous-logs' folder not found on ip-10-0-140-175.ec2.internal, skipping...
2026-07-28T10:59:26.165592359Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T10:59:26.165665986Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T10:59:26.173846090Z Wrote inspect data to must-gather.
2026-07-28T10:59:26.268325583Z Wrote inspect data to must-gather.
2026-07-28T10:59:26.291243317Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T10:59:26.355886844Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T10:59:26.609607489Z INFO: INTERCONNECT MODE
2026-07-28T10:59:26.609640633Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T10:59:26.617439038Z Wrote inspect data to must-gather.
2026-07-28T10:59:26.701231195Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T10:59:26.732161473Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T10:59:26.796967158Z INFO: Gathering OVN_Northbound from ovnkube-node-2bbtd...
2026-07-28T10:59:26.797220304Z INFO: Gathering OVN_Northbound from ovnkube-node-9cmbd...
2026-07-28T10:59:26.797466672Z INFO: Gathering OVN_Northbound from ovnkube-node-nsv94...
2026-07-28T10:59:26.797736802Z INFO: Gathering OVN_Southbound from ovnkube-node-2bbtd...
2026-07-28T10:59:26.798054716Z INFO: Gathering OVN_Southbound from ovnkube-node-9cmbd...
2026-07-28T10:59:26.798338353Z INFO: Gathering OVN_Southbound from ovnkube-node-nsv94...
2026-07-28T10:59:26.891473203Z Wrote inspect data to must-gather.
2026-07-28T10:59:27.044259848Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T10:59:27.108278408Z tar: Removing leading `/' from member names
2026-07-28T10:59:27.112818648Z tar: Removing leading `/' from member names
2026-07-28T10:59:27.118863088Z tar: Removing leading `/' from member names
2026-07-28T10:59:27.133478371Z tar: Removing leading `/' from member names
2026-07-28T10:59:27.148631809Z tar: Removing leading `/' from member names
2026-07-28T10:59:27.219827306Z tar: Removing leading `/' from member names
2026-07-28T10:59:27.258853904Z Wrote inspect data to must-gather.
2026-07-28T10:59:27.261896631Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T10:59:27.346027460Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T10:59:27.528557643Z Wrote inspect data to must-gather.
2026-07-28T10:59:27.599291141Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T10:59:27.630557085Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T10:59:27.692308614Z INFO: Waiting for network log collection to complete ...
2026-07-28T10:59:27.692360564Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T10:59:27.692889321Z INFO: Copying ovnk databases complete.
2026-07-28T10:59:27.694948097Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-07-28T10:59:27.697695251Z ovnk_database_store/
2026-07-28T10:59:27.697695251Z ovnk_database_store/ovnkube-node-2bbtd_nbdb
2026-07-28T10:59:27.714813155Z ovnk_database_store/ovnkube-node-9cmbd_nbdb
2026-07-28T10:59:27.725928333Z ovnk_database_store/ovnkube-node-2bbtd_sbdb
2026-07-28T10:59:27.751918872Z ovnk_database_store/ovnkube-node-9cmbd_sbdb
2026-07-28T10:59:27.778579083Z ovnk_database_store/ovnkube-node-nsv94_nbdb
2026-07-28T10:59:27.785147223Z Wrote inspect data to must-gather.
2026-07-28T10:59:27.788047730Z ovnk_database_store/ovnkube-node-nsv94_sbdb
2026-07-28T10:59:27.815158412Z INFO: Network log collection complete.
2026-07-28T10:59:27.824788838Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:27.935554638Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T10:59:28.051254929Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:28.054550661Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T10:59:28.066983957Z INFO: Getting status from alertmanager-main-0
2026-07-28T10:59:28.112992252Z Wrote inspect data to must-gather.
2026-07-28T10:59:28.145104614Z Gathering data for ns/openshift-network-console...
2026-07-28T10:59:28.281366920Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:28.421481675Z Wrote inspect data to must-gather.
2026-07-28T10:59:28.492823538Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:28.495119664Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T10:59:28.620054580Z Gathering data for ns/openshift-console-operator...
2026-07-28T10:59:28.713857703Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:28.717812377Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T10:59:28.720494877Z INFO: Fallback to identify the container image from release info
2026-07-28T10:59:28.921467791Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:28.924037711Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T10:59:29.030491178Z Gathering data for ns/openshift-console...
2026-07-28T10:59:29.158497766Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:29.214268039Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T10:59:29.339938468Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T10:59:29.366259149Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:29.369607473Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T10:59:29.498570250Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T10:59:29.578537999Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:29.616261584Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T10:59:29.793444707Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:29.795888882Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T10:59:29.983744392Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:30.174150209Z Gathering data for ns/openshift-dns-operator...
2026-07-28T10:59:30.183459516Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:30.187206700Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T10:59:30.217671662Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T10:59:30.358402462Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:30.495632557Z Gathering data for ns/openshift-dns...
2026-07-28T10:59:30.553680617Z Wrote inspect data to /must-gather/istio.
2026-07-28T10:59:30.624300134Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T10:59:31.176236903Z Gathering data for ns/openshift-image-registry...
2026-07-28T10:59:31.227318403Z Done executing Istio gather script
2026-07-28T10:59:31.699078962Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T10:59:31.759324396Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T10:59:32.079829296Z Gathering data for ns/openshift-ingress...
2026-07-28T10:59:32.427512120Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T10:59:32.881085975Z Gathering data for ns/openshift-insights...
2026-07-28T10:59:32.887336130Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T10:59:33.998256209Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T10:59:35.125488216Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T10:59:35.223056865Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T10:59:35.618424047Z Gathering data for ns/kserve...
2026-07-28T10:59:36.238037286Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T10:59:37.279379345Z Gathering data for ns/openshift-config...
2026-07-28T10:59:37.371421124Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T10:59:37.623877212Z Collecting performance related data for node ip-10-0-140-175.ec2.internal
2026-07-28T10:59:37.632850289Z Collecting performance related data for node ip-10-0-131-51.ec2.internal
2026-07-28T10:59:37.641652341Z Collecting performance related data for node ip-10-0-130-25.ec2.internal
2026-07-28T10:59:37.645145160Z Gathering data for ns/openshift-config-managed...
2026-07-28T10:59:38.296565826Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T10:59:38.755701178Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T10:59:39.157972945Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T10:59:39.584857665Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T10:59:40.005134330Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T10:59:40.228620413Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T10:59:40.361739065Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T10:59:40.733331828Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T10:59:41.146695225Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T10:59:41.513895731Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T10:59:42.068026774Z Gathering data for ns/openshift-multus...
2026-07-28T10:59:43.782902518Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T10:59:45.130526352Z Gathering data for ns/openshift-host-network...
2026-07-28T10:59:45.234456086Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T10:59:45.512502434Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T10:59:45.954400848Z Collecting kubelet logs for node ip-10-0-140-175.ec2.internal
2026-07-28T10:59:45.956121112Z Collecting kubelet logs for node ip-10-0-131-51.ec2.internal
2026-07-28T10:59:45.957873609Z Collecting kubelet logs for node ip-10-0-130-25.ec2.internal
2026-07-28T10:59:46.215841938Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T10:59:46.352618515Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-tvln5 namespace
2026-07-28T10:59:46.357117801Z INFO: Node performance data collection complete.
2026-07-28T10:59:46.708462091Z Gathering data for ns/openshift-network-operator...
2026-07-28T10:59:47.128953557Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T10:59:47.477965459Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T10:59:47.979535202Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T10:59:48.291615765Z Gathering data for ns/openshift-apiserver...
2026-07-28T10:59:48.740836170Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T10:59:49.100116136Z Gathering data for ns/openshift-controller-manager...
2026-07-28T10:59:49.418458778Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T10:59:50.129194741Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T10:59:50.239331099Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T10:59:50.455043784Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T10:59:50.802819988Z Gathering data for ns/openshift-service-ca...
2026-07-28T10:59:51.226204163Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T10:59:52.219102685Z Wrote inspect data to must-gather.
2026-07-28T10:59:52.219148514Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T10:59:52.219148514Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T10:59:52.353395603Z error: the server doesn't have a resource type "clusters"
2026-07-28T10:59:52.485903020Z Caches written to disk