2026-11-04T09:56:31.163832896Z [disk usage checker] Started
2026-11-04T09:56:31.167185347Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:56:32.099265114Z Gathering data for ns/openshift-cluster-version...
2026-11-04T09:56:33.000656252Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T09:56:33.557478934Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T09:56:34.063977827Z Gathering data for ns/default...
2026-11-04T09:56:35.930023120Z Gathering data for ns/openshift...
2026-11-04T09:56:36.171881696Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:56:37.900310037Z Gathering data for ns/kube-system...
2026-11-04T09:56:38.121067307Z Waiting on subprocesses to finish execution.
2026-11-04T09:56:38.126275145Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-04T09:56:38.129211012Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-04T09:56:38.131992936Z Executing Istio gather script
2026-11-04T09:56:38.132604087Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-04T09:56:38.142274662Z INFO: Gathering HAProxy config files
2026-11-04T09:56:38.145759522Z INFO: Collecting host service logs for crio
2026-11-04T09:56:38.146201806Z INFO: Collecting host service logs for kubelet
2026-11-04T09:56:38.146615869Z INFO: Collecting host service logs for rpm-ostreed
2026-11-04T09:56:38.146981610Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-04T09:56:38.147355186Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-04T09:56:38.147785589Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-04T09:56:38.148308877Z INFO: Collecting host service logs for NetworkManager
2026-11-04T09:56:38.148720417Z INFO: Collecting host service logs for openvswitch
2026-11-04T09:56:38.149090621Z INFO: Collecting host service logs for ovs-configuration
2026-11-04T09:56:38.149441780Z INFO: Collecting host service logs for ovsdb-server
2026-11-04T09:56:38.149871277Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-04T09:56:38.150354519Z INFO: Waiting for worker host service log collection to complete ...
2026-11-04T09:56:38.188238174Z INFO: Waiting for node performance related collection to complete ...
2026-11-04T09:56:39.115285968Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-lgsrq
2026-11-04T09:56:39.115349051Z insights-runtime-extractor-kj6dg
2026-11-04T09:56:39.115370676Z insights-runtime-extractor-pf8mh
2026-11-04T09:56:39.115390637Z insights-runtime-extractor-r7288
2026-11-04T09:56:39.235805759Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-04T09:56:39.260391899Z Wrote inspect data to must-gather.
2026-11-04T09:56:39.277315126Z No resources found
2026-11-04T09:56:39.332516612Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-04T09:56:39.360706448Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-04T09:56:39.360915327Z INFO: on-disk MachineConfig config collection complete.
2026-11-04T09:56:39.433096825Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-04T09:56:39.433096825Z See 'oc rsync -h' for help and examples
2026-11-04T09:56:39.443266743Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-04T09:56:39.514014154Z INFO: "metallb-operator" not detected. Skipping.
2026-11-04T09:56:39.563507291Z INFO: "nfd" not detected. Skipping.
2026-11-04T09:56:39.565463142Z INFO: Worker host service log collection to complete.
2026-11-04T09:56:39.654254045Z No resources found in openshift-etcd namespace.
2026-11-04T09:56:39.678241173Z error: the server doesn't have a resource type "performanceprofile"
2026-11-04T09:56:39.744013584Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-04T09:56:39.787017078Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-04T09:56:39.842618746Z ERROR: No running kube-apiserver pods found
2026-11-04T09:56:40.207556548Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-04T09:56:40.464635633Z Wrote inspect data to must-gather.
2026-11-04T09:56:40.472168126Z No resources found
2026-11-04T09:56:40.585745662Z INFO: 'previous-logs' folder not found on ip-10-0-130-9.us-west-2.compute.internal, skipping...
2026-11-04T09:56:40.615609030Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T09:56:40.692970221Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-04T09:56:40.883730631Z tar: Removing leading `/' from member names
2026-11-04T09:56:40.924268187Z Wrote inspect data to must-gather.
2026-11-04T09:56:40.942001589Z INFO: HAProxy config collection complete.
2026-11-04T09:56:40.946044571Z error: the server doesn't have a resource type "machineconfigs"
2026-11-04T09:56:41.010503552Z INFO: 'previous-logs' folder not found on ip-10-0-138-204.us-west-2.compute.internal, skipping...
2026-11-04T09:56:41.039855163Z Wrote inspect data to must-gather.
2026-11-04T09:56:41.085074903Z Wrote inspect data to must-gather.
2026-11-04T09:56:41.086415913Z Wrote inspect data to must-gather.
2026-11-04T09:56:41.161733261Z Wrote inspect data to must-gather.
2026-11-04T09:56:41.176950587Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:56:41.319044495Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-04T09:56:41.393005889Z INFO: 'previous-logs' folder not found on ip-10-0-140-90.us-west-2.compute.internal, skipping...
2026-11-04T09:56:41.393047022Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-04T09:56:41.393047022Z INFO: Machine Config Daemon termination log collection complete.
2026-11-04T09:56:41.466769186Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-04T09:56:41.863046266Z Wrote inspect data to must-gather.
2026-11-04T09:56:41.888939942Z INFO: Getting rules from prometheus-k8s-0
2026-11-04T09:56:42.026900431Z Wrote inspect data to must-gather.
2026-11-04T09:56:42.231277774Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-04T09:56:42.373892751Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-04T09:56:42.571424838Z Wrote inspect data to must-gather.
2026-11-04T09:56:42.744237419Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:42.763133298Z Wrote inspect data to must-gather.
2026-11-04T09:56:43.078208188Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-04T09:56:43.248274347Z Wrote inspect data to must-gather.
2026-11-04T09:56:43.316516155Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:43.320194421Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-04T09:56:43.333965156Z Wrote inspect data to must-gather.
2026-11-04T09:56:43.527120440Z INFO: Getting status/config from prometheus-k8s-0
2026-11-04T09:56:43.756027410Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:43.756573995Z Wrote inspect data to must-gather.
2026-11-04T09:56:43.939149894Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T09:56:43.972543581Z INFO: LEGACY MODE
2026-11-04T09:56:43.972602617Z INFO: Gathering ovn-kubernetes master data
2026-11-04T09:56:44.154346746Z Wrote inspect data to must-gather.
2026-11-04T09:56:44.406729027Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:44.409359500Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-04T09:56:44.588709083Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T09:56:44.589822329Z Wrote inspect data to must-gather.
2026-11-04T09:56:44.952431489Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:45.014539976Z Wrote inspect data to must-gather.
2026-11-04T09:56:45.065093781Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-04T09:56:45.534707998Z Wrote inspect data to must-gather.
2026-11-04T09:56:45.642084534Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:45.645867531Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-04T09:56:45.967172733Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-04T09:56:46.018684761Z INFO: Waiting for network log collection to complete ...
2026-11-04T09:56:46.018719644Z INFO: Waiting for ovnk database copies to complete ...
2026-11-04T09:56:46.019125680Z INFO: Copying ovnk databases complete.
2026-11-04T09:56:46.020779158Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-04T09:56:46.021281602Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 802 is not a child of this shell
2026-11-04T09:56:46.021281602Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 803 is not a child of this shell
2026-11-04T09:56:46.021281602Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 804 is not a child of this shell
2026-11-04T09:56:46.021300930Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 805 is not a child of this shell
2026-11-04T09:56:46.021300930Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 806 is not a child of this shell
2026-11-04T09:56:46.021300930Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 807 is not a child of this shell
2026-11-04T09:56:46.021300930Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 808 is not a child of this shell
2026-11-04T09:56:46.021300930Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 809 is not a child of this shell
2026-11-04T09:56:46.021300930Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 817 is not a child of this shell
2026-11-04T09:56:46.021312872Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 818 is not a child of this shell
2026-11-04T09:56:46.021312872Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 819 is not a child of this shell
2026-11-04T09:56:46.021312872Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 820 is not a child of this shell
2026-11-04T09:56:46.021322826Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 821 is not a child of this shell
2026-11-04T09:56:46.026085899Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-04T09:56:46.029976467Z tar: Exiting with failure status due to previous errors
2026-11-04T09:56:46.030482334Z INFO: Network log collection complete.
2026-11-04T09:56:46.064503904Z Gathering data for ns/openshift-etcd...
2026-11-04T09:56:46.183552664Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:56:46.261176104Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:46.288701831Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-04T09:56:46.288701831Z Gathering data for ns/openshift-monitoring...
2026-11-04T09:56:46.667733763Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-04T09:56:46.848458226Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-04T09:56:46.909184326Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:46.912253844Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-04T09:56:47.034816811Z daemonset.apps/perf-node-gather-daemonset created
2026-11-04T09:56:47.290722976Z Waiting for performance profile collector pods to become ready: 1
2026-11-04T09:56:47.490092334Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:48.004676422Z Wrote inspect data to must-gather.
2026-11-04T09:56:48.004715103Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T09:56:48.004715103Z     namespaces "assisted-installer" not found
2026-11-04T09:56:48.009945818Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-04T09:56:48.149497860Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:48.152412544Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-04T09:56:48.558550637Z Waiting for performance profile collector pods to become ready: 2
2026-11-04T09:56:48.595183249Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:49.228950210Z Wrote inspect data to /must-gather/istio.
2026-11-04T09:56:49.255558465Z INFO: Getting status from alertmanager-main-0
2026-11-04T09:56:49.814408682Z Waiting for performance profile collector pods to become ready: 3
2026-11-04T09:56:51.078845658Z Waiting for performance profile collector pods to become ready: 4
2026-11-04T09:56:51.147431583Z Done executing Istio gather script
2026-11-04T09:56:51.188549482Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:56:52.349038219Z Waiting for performance profile collector pods to become ready: 5
2026-11-04T09:56:53.601996015Z Waiting for performance profile collector pods to become ready: 6
2026-11-04T09:56:54.858607690Z Waiting for performance profile collector pods to become ready: 7
2026-11-04T09:56:56.114958041Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-04T09:56:56.193885529Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:56:56.672457187Z Collecting performance related data for node ip-10-0-140-90.us-west-2.compute.internal
2026-11-04T09:56:56.679447528Z Collecting performance related data for node ip-10-0-130-9.us-west-2.compute.internal
2026-11-04T09:56:56.687164799Z Collecting performance related data for node ip-10-0-138-204.us-west-2.compute.internal
2026-11-04T09:57:01.198977944Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:57:06.203659944Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:57:08.562681701Z Gathering data for ns/openshift-network-console...
2026-11-04T09:57:10.508024979Z Collecting kubelet logs for node ip-10-0-140-90.us-west-2.compute.internal
2026-11-04T09:57:10.509859789Z Collecting kubelet logs for node ip-10-0-130-9.us-west-2.compute.internal
2026-11-04T09:57:10.512177132Z Collecting kubelet logs for node ip-10-0-138-204.us-west-2.compute.internal
2026-11-04T09:57:11.208409522Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:57:11.417502568Z Gathering data for ns/openshift-console-operator...
2026-11-04T09:57:11.427039701Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-x5jj2 namespace
2026-11-04T09:57:11.484857526Z INFO: Node performance data collection complete.
2026-11-04T09:57:13.786142492Z Gathering data for ns/openshift-console...
2026-11-04T09:57:16.213121828Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:57:17.183012108Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-04T09:57:20.563018873Z Gathering data for ns/openshift-dns-operator...
2026-11-04T09:57:21.217811579Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-04T09:57:22.798957413Z Gathering data for ns/openshift-dns...
2026-11-04T09:57:26.222411689Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:57:29.299440423Z Gathering data for ns/openshift-image-registry...
2026-11-04T09:57:31.227089417Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:57:33.210479880Z Gathering data for ns/openshift-ingress-operator...
2026-11-04T09:57:35.356466173Z Gathering data for ns/openshift-ingress...
2026-11-04T09:57:36.231756158Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:57:38.000679824Z Gathering data for ns/openshift-ingress-canary...
2026-11-04T09:57:41.236328814Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:57:41.506530923Z Gathering data for ns/openshift-insights...
2026-11-04T09:57:46.241036097Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:57:50.157011539Z Gathering data for ns/openshift-jobset-operator...
2026-11-04T09:57:51.245744500Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:57:53.517512314Z Gathering data for ns/openshift-config...
2026-11-04T09:57:55.490340418Z Gathering data for ns/openshift-config-managed...
2026-11-04T09:57:56.250607190Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:57:57.650340809Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-04T09:57:59.622928628Z Gathering data for ns/openshift-kube-apiserver...
2026-11-04T09:58:01.255243221Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:01.598228961Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-04T09:58:03.597892085Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-04T09:58:05.713545370Z Gathering data for ns/openshift-kube-scheduler...
2026-11-04T09:58:06.259871968Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:07.685514945Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-04T09:58:09.799757195Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-04T09:58:11.264504749Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:12.684713425Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-04T09:58:15.114476828Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-04T09:58:16.269144867Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:18.028357021Z Gathering data for ns/openshift-multus...
2026-11-04T09:58:21.273970221Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:26.278600244Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:31.283197818Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:35.960062691Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-04T09:58:36.287938871Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:41.292694558Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:46.297368798Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:50.610328096Z Gathering data for ns/openshift-host-network...
2026-11-04T09:58:51.302406298Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:52.905509450Z Gathering data for ns/openshift-network-diagnostics...
2026-11-04T09:58:56.307121252Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:58:57.919641512Z Gathering data for ns/openshift-network-node-identity...
2026-11-04T09:59:01.273691652Z Gathering data for ns/openshift-network-operator...
2026-11-04T09:59:01.311746331Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:04.738396434Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-04T09:59:06.316407671Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:06.748733587Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-04T09:59:10.640730606Z Gathering data for ns/openshift-apiserver-operator...
2026-11-04T09:59:11.320920124Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:12.665724009Z Gathering data for ns/openshift-apiserver...
2026-11-04T09:59:15.466514686Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-04T09:59:16.325734914Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:17.470954520Z Gathering data for ns/openshift-controller-manager...
2026-11-04T09:59:19.542970970Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-04T09:59:21.330482162Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:22.914998138Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-04T09:59:24.995986392Z Gathering data for ns/openshift-service-ca-operator...
2026-11-04T09:59:26.335229363Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:27.462685561Z Gathering data for ns/openshift-service-ca...
2026-11-04T09:59:30.330618845Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-04T09:59:31.339882778Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:36.347246952Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-11-04T09:59:36.902957065Z Wrote inspect data to must-gather.
2026-11-04T09:59:36.903002278Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T09:59:36.903002278Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-04T09:59:37.372998962Z error: the server doesn't have a resource type "clusters"
2026-11-04T09:59:37.425618255Z Caches written to disk