2026-11-02T03:32:35.310609113Z [disk usage checker] Started
2026-11-02T03:32:35.313912554Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:32:36.242023215Z Gathering data for ns/openshift-cluster-version...
2026-11-02T03:32:36.974489326Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-02T03:32:37.495265063Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-02T03:32:37.972197339Z Gathering data for ns/default...
2026-11-02T03:32:39.731631650Z Gathering data for ns/openshift...
2026-11-02T03:32:40.318877626Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:32:41.587860418Z Gathering data for ns/kube-system...
2026-11-02T03:32:41.970503560Z Waiting on subprocesses to finish execution.
2026-11-02T03:32:41.981712220Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-02T03:32:41.985404960Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-02T03:32:41.989264281Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-02T03:32:41.990596640Z Executing Istio gather script
2026-11-02T03:32:41.990901515Z INFO: Gathering HAProxy config files
2026-11-02T03:32:42.027050388Z INFO: Collecting host service logs for crio
2026-11-02T03:32:42.027328746Z INFO: Collecting host service logs for kubelet
2026-11-02T03:32:42.027792151Z INFO: Collecting host service logs for rpm-ostreed
2026-11-02T03:32:42.028105536Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-02T03:32:42.028433059Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-02T03:32:42.028731755Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-02T03:32:42.029022722Z INFO: Collecting host service logs for NetworkManager
2026-11-02T03:32:42.029383708Z INFO: Collecting host service logs for openvswitch
2026-11-02T03:32:42.029654073Z INFO: Collecting host service logs for ovs-configuration
2026-11-02T03:32:42.029949584Z INFO: Collecting host service logs for ovsdb-server
2026-11-02T03:32:42.030268719Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-02T03:32:42.030686713Z INFO: Waiting for worker host service log collection to complete ...
2026-11-02T03:32:42.333494154Z INFO: Waiting for node performance related collection to complete ...
2026-11-02T03:32:42.613664955Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-r9k59
2026-11-02T03:32:42.613721895Z insights-runtime-extractor-8gg2w
2026-11-02T03:32:42.613742534Z insights-runtime-extractor-jlqtp
2026-11-02T03:32:42.613762363Z insights-runtime-extractor-jw5jb
2026-11-02T03:32:42.808893926Z No resources found in openshift-etcd namespace.
2026-11-02T03:32:42.943011213Z No resources found
2026-11-02T03:32:43.120942217Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-02T03:32:43.127177084Z INFO: "nfd" not detected. Skipping.
2026-11-02T03:32:43.170824784Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-02T03:32:43.177093969Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-02T03:32:43.177142196Z INFO: on-disk MachineConfig config collection complete.
2026-11-02T03:32:43.182989904Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-02T03:32:43.312617670Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-02T03:32:43.430016238Z Wrote inspect data to must-gather.
2026-11-02T03:32:43.445121857Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-02T03:32:43.445121857Z See 'oc rsync -h' for help and examples
2026-11-02T03:32:43.465258102Z INFO: "metallb-operator" not detected. Skipping.
2026-11-02T03:32:43.618792075Z INFO: 'previous-logs' folder not found on ip-10-0-128-119.us-west-1.compute.internal, skipping...
2026-11-02T03:32:43.680326459Z error: the server doesn't have a resource type "performanceprofile"
2026-11-02T03:32:43.682377614Z INFO: Worker host service log collection to complete.
2026-11-02T03:32:43.742078734Z ERROR: No running kube-apiserver pods found
2026-11-02T03:32:43.989693587Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-02T03:32:44.224629679Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-02T03:32:44.238742330Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-02T03:32:44.435202174Z INFO: 'previous-logs' folder not found on ip-10-0-130-20.us-west-1.compute.internal, skipping...
2026-11-02T03:32:44.469754452Z No resources found
2026-11-02T03:32:44.510938500Z Wrote inspect data to must-gather.
2026-11-02T03:32:44.599499111Z Wrote inspect data to must-gather.
2026-11-02T03:32:44.628844309Z Wrote inspect data to must-gather.
2026-11-02T03:32:44.655604001Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-02T03:32:44.724023086Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-02T03:32:44.844177551Z Wrote inspect data to must-gather.
2026-11-02T03:32:44.874093095Z INFO: 'previous-logs' folder not found on ip-10-0-139-39.us-west-1.compute.internal, skipping...
2026-11-02T03:32:44.874126927Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-02T03:32:44.874159793Z INFO: Machine Config Daemon termination log collection complete.
2026-11-02T03:32:44.893700892Z Wrote inspect data to must-gather.
2026-11-02T03:32:44.960624804Z Wrote inspect data to must-gather.
2026-11-02T03:32:44.982008047Z error: the server doesn't have a resource type "machineconfigs"
2026-11-02T03:32:45.295828838Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-02T03:32:45.324082864Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:32:45.324870459Z tar: Removing leading `/' from member names
2026-11-02T03:32:45.378360036Z INFO: HAProxy config collection complete.
2026-11-02T03:32:45.476249227Z Wrote inspect data to must-gather.
2026-11-02T03:32:45.698162013Z Wrote inspect data to must-gather.
2026-11-02T03:32:45.772650101Z INFO: Getting rules from prometheus-k8s-0
2026-11-02T03:32:45.821157545Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-02T03:32:46.019385971Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-02T03:32:46.115905719Z Wrote inspect data to must-gather.
2026-11-02T03:32:46.321868391Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:46.422292525Z Wrote inspect data to must-gather.
2026-11-02T03:32:46.761768162Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-02T03:32:46.775560262Z Wrote inspect data to must-gather.
2026-11-02T03:32:46.827948718Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:46.831237523Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-02T03:32:46.923583589Z Wrote inspect data to must-gather.
2026-11-02T03:32:47.290280821Z INFO: Getting status/config from prometheus-k8s-0
2026-11-02T03:32:47.315875048Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:47.340100274Z Wrote inspect data to must-gather.
2026-11-02T03:32:47.439323695Z INFO: LEGACY MODE
2026-11-02T03:32:47.439371785Z INFO: Gathering ovn-kubernetes master data
2026-11-02T03:32:47.525775211Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-02T03:32:47.759718682Z Wrote inspect data to must-gather.
2026-11-02T03:32:47.944758655Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:47.947403351Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-02T03:32:48.091452953Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-02T03:32:48.163701514Z Wrote inspect data to must-gather.
2026-11-02T03:32:48.445221463Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:48.577369377Z Wrote inspect data to must-gather.
2026-11-02T03:32:48.699181401Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-02T03:32:49.081153886Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:49.085171455Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-02T03:32:49.118412907Z Wrote inspect data to must-gather.
2026-11-02T03:32:49.378649154Z Gathering data for ns/openshift-etcd...
2026-11-02T03:32:49.411933578Z INFO: Waiting for network log collection to complete ...
2026-11-02T03:32:49.411965692Z INFO: Waiting for ovnk database copies to complete ...
2026-11-02T03:32:49.412456894Z INFO: Copying ovnk databases complete.
2026-11-02T03:32:49.414175723Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-02T03:32:49.414726565Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 812 is not a child of this shell
2026-11-02T03:32:49.414726565Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 813 is not a child of this shell
2026-11-02T03:32:49.414726565Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 814 is not a child of this shell
2026-11-02T03:32:49.414726565Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 816 is not a child of this shell
2026-11-02T03:32:49.414726565Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 817 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 818 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 819 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 820 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 823 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 824 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 825 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 827 is not a child of this shell
2026-11-02T03:32:49.414752414Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 828 is not a child of this shell
2026-11-02T03:32:49.418029737Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-02T03:32:49.421844112Z tar: Exiting with failure status due to previous errors
2026-11-02T03:32:49.422410126Z INFO: Network log collection complete.
2026-11-02T03:32:49.582052907Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-02T03:32:49.611448668Z Gathering data for ns/openshift-monitoring...
2026-11-02T03:32:49.651575386Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:49.921387951Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-02T03:32:50.252546172Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:50.255405647Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-02T03:32:50.276329246Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-02T03:32:50.329015530Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:32:50.436547424Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-02T03:32:50.634909124Z daemonset.apps/perf-node-gather-daemonset created
2026-11-02T03:32:50.773652753Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:50.894829046Z Waiting for performance profile collector pods to become ready: 1
2026-11-02T03:32:51.160588337Z Wrote inspect data to must-gather.
2026-11-02T03:32:51.160630569Z error: inspection completed with the errors occurred while gathering data:
2026-11-02T03:32:51.160630569Z     namespaces "assisted-installer" not found
2026-11-02T03:32:51.429015294Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:51.431764337Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-02T03:32:51.619738726Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-02T03:32:51.810388263Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:52.145246991Z Waiting for performance profile collector pods to become ready: 2
2026-11-02T03:32:52.405849069Z Wrote inspect data to /must-gather/istio.
2026-11-02T03:32:52.832919924Z INFO: Getting status from alertmanager-main-0
2026-11-02T03:32:53.393793980Z Waiting for performance profile collector pods to become ready: 3
2026-11-02T03:32:54.109032092Z Done executing Istio gather script
2026-11-02T03:32:54.638767435Z Waiting for performance profile collector pods to become ready: 4
2026-11-02T03:32:55.333768988Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:32:55.894282065Z Waiting for performance profile collector pods to become ready: 5
2026-11-02T03:32:57.141365332Z Waiting for performance profile collector pods to become ready: 6
2026-11-02T03:32:58.388137523Z Waiting for performance profile collector pods to become ready: 7
2026-11-02T03:32:59.639102437Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-02T03:33:00.191608359Z Collecting performance related data for node ip-10-0-128-119.us-west-1.compute.internal
2026-11-02T03:33:00.198735674Z Collecting performance related data for node ip-10-0-139-39.us-west-1.compute.internal
2026-11-02T03:33:00.206389232Z Collecting performance related data for node ip-10-0-130-20.us-west-1.compute.internal
2026-11-02T03:33:00.338421385Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:05.343044673Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:10.348593550Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:10.456086082Z Gathering data for ns/openshift-network-console...
2026-11-02T03:33:13.073365064Z Gathering data for ns/openshift-console-operator...
2026-11-02T03:33:13.539581343Z Collecting kubelet logs for node ip-10-0-128-119.us-west-1.compute.internal
2026-11-02T03:33:13.541537696Z Collecting kubelet logs for node ip-10-0-139-39.us-west-1.compute.internal
2026-11-02T03:33:13.543445324Z Collecting kubelet logs for node ip-10-0-130-20.us-west-1.compute.internal
2026-11-02T03:33:14.402155537Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-b72f8 namespace
2026-11-02T03:33:14.456586644Z INFO: Node performance data collection complete.
2026-11-02T03:33:15.353260937Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:15.359358484Z Gathering data for ns/openshift-console...
2026-11-02T03:33:18.410070997Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-02T03:33:20.358482582Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:21.461322773Z Gathering data for ns/openshift-dns-operator...
2026-11-02T03:33:23.481492709Z Gathering data for ns/openshift-dns...
2026-11-02T03:33:25.363165593Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:29.407864523Z Gathering data for ns/openshift-image-registry...
2026-11-02T03:33:30.367809214Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:32.941999112Z Gathering data for ns/openshift-ingress-operator...
2026-11-02T03:33:34.866909034Z Gathering data for ns/openshift-ingress...
2026-11-02T03:33:35.372836870Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:37.204742624Z Gathering data for ns/openshift-ingress-canary...
2026-11-02T03:33:40.373584021Z Gathering data for ns/openshift-insights...
2026-11-02T03:33:40.377742505Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:45.382779407Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:48.585400353Z Gathering data for ns/openshift-config...
2026-11-02T03:33:50.361429870Z Gathering data for ns/openshift-config-managed...
2026-11-02T03:33:50.387527075Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:52.307004467Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-02T03:33:54.128279240Z Gathering data for ns/openshift-kube-apiserver...
2026-11-02T03:33:55.392345102Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:33:55.905121196Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-02T03:33:57.726831374Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-02T03:33:59.616495945Z Gathering data for ns/openshift-kube-scheduler...
2026-11-02T03:34:00.397628601Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:01.389961323Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-02T03:34:03.284028491Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-02T03:34:05.402412250Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:05.935050163Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-02T03:34:08.146983870Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-02T03:34:10.406936915Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:10.765088187Z Gathering data for ns/openshift-multus...
2026-11-02T03:34:15.412019779Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:20.417445699Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:25.422439604Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:27.197702741Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-02T03:34:30.427265283Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:35.432180884Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:40.436760587Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:40.573363172Z Gathering data for ns/openshift-host-network...
2026-11-02T03:34:42.640224360Z Gathering data for ns/openshift-network-diagnostics...
2026-11-02T03:34:45.441419293Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:47.168227174Z Gathering data for ns/openshift-network-node-identity...
2026-11-02T03:34:50.142018122Z Gathering data for ns/openshift-network-operator...
2026-11-02T03:34:50.446074111Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:53.271032898Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-02T03:34:55.051642233Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-02T03:34:55.450721837Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:34:58.554672234Z Gathering data for ns/openshift-apiserver-operator...
2026-11-02T03:35:00.359180305Z Gathering data for ns/openshift-apiserver...
2026-11-02T03:35:00.455370827Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:35:02.839626965Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-02T03:35:04.641282548Z Gathering data for ns/openshift-controller-manager...
2026-11-02T03:35:05.460477292Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:35:06.480476735Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-02T03:35:09.551174976Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-02T03:35:10.465712987Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:35:11.391595179Z Gathering data for ns/openshift-service-ca-operator...
2026-11-02T03:35:13.607435716Z Gathering data for ns/openshift-service-ca...
2026-11-02T03:35:15.470335876Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:35:16.158585224Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-02T03:35:20.475225126Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-02T03:35:22.142742360Z Wrote inspect data to must-gather.
2026-11-02T03:35:22.142793052Z error: inspection completed with the errors occurred while gathering data:
2026-11-02T03:35:22.142793052Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-02T03:35:22.560634358Z error: the server doesn't have a resource type "clusters"
2026-11-02T03:35:22.618464894Z Caches written to disk