2026-11-04T08:31:41.520033443Z [disk usage checker] Started
2026-11-04T08:31:41.523389116Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:31:42.467947306Z Gathering data for ns/openshift-cluster-version...
2026-11-04T08:31:43.394061782Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T08:31:43.973343385Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T08:31:44.487815669Z Gathering data for ns/default...
2026-11-04T08:31:46.425263746Z Gathering data for ns/openshift...
2026-11-04T08:31:46.528857649Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:31:48.491636651Z Gathering data for ns/kube-system...
2026-11-04T08:31:48.543038670Z Waiting on subprocesses to finish execution.
2026-11-04T08:31:48.546676579Z INFO: Gathering HAProxy config files
2026-11-04T08:31:48.553300046Z Executing Istio gather script
2026-11-04T08:31:48.563838496Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-04T08:31:48.575918190Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-04T08:31:48.593930242Z INFO: Waiting for node performance related collection to complete ...
2026-11-04T08:31:48.603126544Z INFO: Collecting host service logs for crio
2026-11-04T08:31:48.603163950Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-04T08:31:48.603548663Z INFO: Collecting host service logs for kubelet
2026-11-04T08:31:48.603928737Z INFO: Collecting host service logs for rpm-ostreed
2026-11-04T08:31:48.604297677Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-04T08:31:48.604644517Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-04T08:31:48.604977540Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-04T08:31:48.605293842Z INFO: Collecting host service logs for NetworkManager
2026-11-04T08:31:48.605604530Z INFO: Collecting host service logs for openvswitch
2026-11-04T08:31:48.605890634Z INFO: Collecting host service logs for ovs-configuration
2026-11-04T08:31:48.606194058Z INFO: Collecting host service logs for ovsdb-server
2026-11-04T08:31:48.606466529Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-04T08:31:48.606828155Z INFO: Waiting for worker host service log collection to complete ...
2026-11-04T08:31:49.286672434Z INFO: "metallb-operator" not detected. Skipping.
2026-11-04T08:31:49.310099242Z No resources found in openshift-etcd namespace.
2026-11-04T08:31:49.439980785Z INFO: "nfd" not detected. Skipping.
2026-11-04T08:31:49.499764169Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-04T08:31:49.499801085Z INFO: on-disk MachineConfig config collection complete.
2026-11-04T08:31:49.540561962Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-hxmgz
2026-11-04T08:31:49.540628184Z insights-runtime-extractor-6wnz5
2026-11-04T08:31:49.540651810Z insights-runtime-extractor-tkskc
2026-11-04T08:31:49.540683363Z insights-runtime-extractor-zqhr6
2026-11-04T08:31:49.827111681Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-04T08:31:49.926888846Z Wrote inspect data to must-gather.
2026-11-04T08:31:49.959617347Z error: the server doesn't have a resource type "performanceprofile"
2026-11-04T08:31:50.013849510Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-04T08:31:50.046225514Z ERROR: No running kube-apiserver pods found
2026-11-04T08:31:50.075345479Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-04T08:31:50.075345479Z See 'oc rsync -h' for help and examples
2026-11-04T08:31:50.079386229Z INFO: Worker host service log collection to complete.
2026-11-04T08:31:50.184719052Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-04T08:31:50.190698022Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-04T08:31:50.194179142Z No resources found
2026-11-04T08:31:50.361803177Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-04T08:31:50.746785208Z Wrote inspect data to must-gather.
2026-11-04T08:31:50.774551657Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-04T08:31:50.814126821Z No resources found
2026-11-04T08:31:51.023552639Z Wrote inspect data to must-gather.
2026-11-04T08:31:51.102224098Z INFO: 'previous-logs' folder not found on ip-10-0-130-116.us-west-1.compute.internal, skipping...
2026-11-04T08:31:51.136083587Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T08:31:51.163922711Z Wrote inspect data to must-gather.
2026-11-04T08:31:51.216788857Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-04T08:31:51.274905776Z error: the server doesn't have a resource type "machineconfigs"
2026-11-04T08:31:51.400130308Z Wrote inspect data to must-gather.
2026-11-04T08:31:51.483044026Z Wrote inspect data to must-gather.
2026-11-04T08:31:51.486866908Z Wrote inspect data to must-gather.
2026-11-04T08:31:51.527085313Z INFO: 'previous-logs' folder not found on ip-10-0-135-73.us-west-1.compute.internal, skipping...
2026-11-04T08:31:51.533930006Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:31:51.588564762Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-04T08:31:51.737295254Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-04T08:31:51.787151631Z tar: Removing leading `/' from member names
2026-11-04T08:31:51.890200148Z INFO: HAProxy config collection complete.
2026-11-04T08:31:51.919070909Z INFO: 'previous-logs' folder not found on ip-10-0-136-49.us-west-1.compute.internal, skipping...
2026-11-04T08:31:51.919121629Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-04T08:31:51.919121629Z INFO: Machine Config Daemon termination log collection complete.
2026-11-04T08:31:52.191707883Z Wrote inspect data to must-gather.
2026-11-04T08:31:52.294642584Z Wrote inspect data to must-gather.
2026-11-04T08:31:52.327191544Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-04T08:31:52.369928518Z INFO: Getting rules from prometheus-k8s-0
2026-11-04T08:31:52.750886571Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-04T08:31:52.866209348Z Wrote inspect data to must-gather.
2026-11-04T08:31:52.903447737Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:53.184168625Z Wrote inspect data to must-gather.
2026-11-04T08:31:53.545363888Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:53.548302150Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-04T08:31:53.579235048Z Wrote inspect data to must-gather.
2026-11-04T08:31:53.651428719Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-04T08:31:53.718850864Z Wrote inspect data to must-gather.
2026-11-04T08:31:54.018760038Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:54.047146895Z INFO: Getting status/config from prometheus-k8s-0
2026-11-04T08:31:54.180284739Z Wrote inspect data to must-gather.
2026-11-04T08:31:54.293100851Z INFO: LEGACY MODE
2026-11-04T08:31:54.293144412Z INFO: Gathering ovn-kubernetes master data
2026-11-04T08:31:54.472553515Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T08:31:54.598065370Z Wrote inspect data to must-gather.
2026-11-04T08:31:54.705477237Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:54.708150227Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-04T08:31:55.026158968Z Wrote inspect data to must-gather.
2026-11-04T08:31:55.086025462Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T08:31:55.294140518Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:55.470982785Z Wrote inspect data to must-gather.
2026-11-04T08:31:55.537686485Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-04T08:31:55.964534814Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:55.967558045Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-04T08:31:56.084355541Z Wrote inspect data to must-gather.
2026-11-04T08:31:56.396996204Z INFO: Waiting for network log collection to complete ...
2026-11-04T08:31:56.397036130Z INFO: Waiting for ovnk database copies to complete ...
2026-11-04T08:31:56.397415799Z INFO: Copying ovnk databases complete.
2026-11-04T08:31:56.399567242Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 792 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 793 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 795 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 797 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 798 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 800 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 803 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 805 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 835 is not a child of this shell
2026-11-04T08:31:56.400926909Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 836 is not a child of this shell
2026-11-04T08:31:56.400956222Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 837 is not a child of this shell
2026-11-04T08:31:56.400956222Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 839 is not a child of this shell
2026-11-04T08:31:56.400956222Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 840 is not a child of this shell
2026-11-04T08:31:56.406261461Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-04T08:31:56.410060307Z tar: Exiting with failure status due to previous errors
2026-11-04T08:31:56.410642034Z INFO: Network log collection complete.
2026-11-04T08:31:56.482723023Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-04T08:31:56.539230443Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:31:56.560214094Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:56.760109890Z Gathering data for ns/openshift-monitoring...
2026-11-04T08:31:56.950784849Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-04T08:31:57.272923971Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:57.275640094Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-04T08:31:57.335380897Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-04T08:31:57.379620493Z Gathering data for ns/openshift-etcd...
2026-11-04T08:31:57.424512971Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-04T08:31:57.724800670Z daemonset.apps/perf-node-gather-daemonset created
2026-11-04T08:31:57.830617568Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:57.994398575Z Waiting for performance profile collector pods to become ready: 1
2026-11-04T08:31:58.480009740Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:58.482800657Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-04T08:31:58.685427222Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-04T08:31:58.942129872Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:59.274518810Z Waiting for performance profile collector pods to become ready: 2
2026-11-04T08:31:59.314755294Z Wrote inspect data to must-gather.
2026-11-04T08:31:59.314799430Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T08:31:59.314799430Z     namespaces "assisted-installer" not found
2026-11-04T08:31:59.596280762Z Wrote inspect data to /must-gather/istio.
2026-11-04T08:31:59.980947580Z INFO: Getting status from alertmanager-main-0
2026-11-04T08:32:00.536912325Z Waiting for performance profile collector pods to become ready: 3
2026-11-04T08:32:01.430446093Z Done executing Istio gather script
2026-11-04T08:32:01.543992488Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:01.796434376Z Waiting for performance profile collector pods to become ready: 4
2026-11-04T08:32:03.059964440Z Waiting for performance profile collector pods to become ready: 5
2026-11-04T08:32:04.320019833Z Waiting for performance profile collector pods to become ready: 6
2026-11-04T08:32:05.584559455Z Waiting for performance profile collector pods to become ready: 7
2026-11-04T08:32:06.548670785Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:06.841818321Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-04T08:32:07.425142509Z Collecting performance related data for node ip-10-0-130-116.us-west-1.compute.internal
2026-11-04T08:32:07.432184745Z Collecting performance related data for node ip-10-0-136-49.us-west-1.compute.internal
2026-11-04T08:32:07.439666613Z Collecting performance related data for node ip-10-0-135-73.us-west-1.compute.internal
2026-11-04T08:32:11.556755203Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:16.561911187Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:19.154128767Z Gathering data for ns/openshift-network-console...
2026-11-04T08:32:21.566859527Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:21.568867505Z Collecting kubelet logs for node ip-10-0-130-116.us-west-1.compute.internal
2026-11-04T08:32:21.570662679Z Collecting kubelet logs for node ip-10-0-136-49.us-west-1.compute.internal
2026-11-04T08:32:21.572992301Z Collecting kubelet logs for node ip-10-0-135-73.us-west-1.compute.internal
2026-11-04T08:32:22.002346039Z Gathering data for ns/openshift-console-operator...
2026-11-04T08:32:22.543975988Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-z9d4m namespace
2026-11-04T08:32:22.602441968Z INFO: Node performance data collection complete.
2026-11-04T08:32:24.379815917Z Gathering data for ns/openshift-console...
2026-11-04T08:32:26.571512797Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:27.679629508Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-04T08:32:30.988992560Z Gathering data for ns/openshift-dns-operator...
2026-11-04T08:32:31.576204374Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:33.169034323Z Gathering data for ns/openshift-dns...
2026-11-04T08:32:36.580779671Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:39.604050609Z Gathering data for ns/openshift-image-registry...
2026-11-04T08:32:41.585542016Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:43.540468543Z Gathering data for ns/openshift-ingress-operator...
2026-11-04T08:32:45.614281935Z Gathering data for ns/openshift-ingress...
2026-11-04T08:32:46.590214277Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:48.161804896Z Gathering data for ns/openshift-ingress-canary...
2026-11-04T08:32:51.591578697Z Gathering data for ns/openshift-insights...
2026-11-04T08:32:51.595022205Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:32:56.599668003Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:00.088454423Z Gathering data for ns/openshift-jobset-operator...
2026-11-04T08:33:01.604417741Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:03.288538097Z Gathering data for ns/openshift-config...
2026-11-04T08:33:05.201768864Z Gathering data for ns/openshift-config-managed...
2026-11-04T08:33:06.609142559Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:07.313124220Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-04T08:33:09.229439963Z Gathering data for ns/openshift-kube-apiserver...
2026-11-04T08:33:11.142651828Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-04T08:33:11.613975763Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:13.081142171Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-04T08:33:15.124859795Z Gathering data for ns/openshift-kube-scheduler...
2026-11-04T08:33:16.618621866Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:17.039565063Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-04T08:33:19.076783272Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-04T08:33:21.623878022Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:21.941011995Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-04T08:33:24.337248860Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-04T08:33:26.628825950Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:27.163653057Z Gathering data for ns/openshift-multus...
2026-11-04T08:33:31.633435133Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:36.638072531Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:41.642752619Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:44.866381132Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-04T08:33:46.647446723Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:51.652101071Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:56.657000161Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:33:59.265238660Z Gathering data for ns/openshift-host-network...
2026-11-04T08:34:01.504103784Z Gathering data for ns/openshift-network-diagnostics...
2026-11-04T08:34:01.661704965Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:06.369807496Z Gathering data for ns/openshift-network-node-identity...
2026-11-04T08:34:06.666312992Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:09.571104666Z Gathering data for ns/openshift-network-operator...
2026-11-04T08:34:11.671022568Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:12.937920719Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-04T08:34:14.858526369Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-04T08:34:16.675721850Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:18.669036104Z Gathering data for ns/openshift-apiserver-operator...
2026-11-04T08:34:20.608105312Z Gathering data for ns/openshift-apiserver...
2026-11-04T08:34:21.680372325Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:23.276734904Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-04T08:34:25.198464393Z Gathering data for ns/openshift-controller-manager...
2026-11-04T08:34:26.685064044Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:27.188820623Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-04T08:34:30.492312859Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-04T08:34:31.689688284Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:32.480263795Z Gathering data for ns/openshift-service-ca-operator...
2026-11-04T08:34:34.871755977Z Gathering data for ns/openshift-service-ca...
2026-11-04T08:34:36.694334576Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:37.652828464Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-04T08:34:41.699045544Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T08:34:44.157835846Z Wrote inspect data to must-gather.
2026-11-04T08:34:44.157889186Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T08:34:44.157889186Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-04T08:34:44.591893209Z error: the server doesn't have a resource type "clusters"
2026-11-04T08:34:44.641559257Z Caches written to disk