2026-11-04T04:10:51.089623383Z [disk usage checker] Started 2026-11-04T04:10:51.093157276Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:10:51.945767579Z Gathering data for ns/openshift-cluster-version... 2026-11-04T04:10:52.657395085Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-11-04T04:10:53.217783440Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-11-04T04:10:53.688021771Z Gathering data for ns/default... 2026-11-04T04:10:55.410596870Z Gathering data for ns/openshift... 2026-11-04T04:10:56.097475922Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:10:57.247657522Z Gathering data for ns/kube-system... 2026-11-04T04:10:58.129213169Z Waiting on subprocesses to finish execution. 2026-11-04T04:10:58.139343791Z INFO: Gathering machine config daemon's old logs from all nodes 2026-11-04T04:10:58.142867297Z Executing Istio gather script 2026-11-04T04:10:58.149296594Z INFO: Waiting for node performance related collection to complete ... 2026-11-04T04:10:58.151094899Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-11-04T04:10:58.154686221Z INFO: Collecting host service logs for crio 2026-11-04T04:10:58.155164389Z INFO: Collecting host service logs for kubelet 2026-11-04T04:10:58.155494901Z INFO: Collecting host service logs for rpm-ostreed 2026-11-04T04:10:58.155818556Z INFO: Collecting host service logs for ostree-finalize-staged 2026-11-04T04:10:58.156231236Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-11-04T04:10:58.156581809Z INFO: Collecting host service logs for machine-config-daemon-host 2026-11-04T04:10:58.156932166Z INFO: Collecting host service logs for NetworkManager 2026-11-04T04:10:58.157254685Z INFO: Collecting host service logs for openvswitch 2026-11-04T04:10:58.157571695Z INFO: Collecting host service logs for ovs-configuration 2026-11-04T04:10:58.157897727Z INFO: Collecting host service logs for ovsdb-server 2026-11-04T04:10:58.158258140Z INFO: Collecting host service logs for ovs-vswitchd 2026-11-04T04:10:58.158641299Z INFO: Waiting for worker host service log collection to complete ... 2026-11-04T04:10:58.158793864Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-11-04T04:10:58.172056793Z INFO: Gathering HAProxy config files 2026-11-04T04:10:58.865549186Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-11-04T04:10:59.011986180Z No resources found 2026-11-04T04:10:59.015467964Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-11-04T04:10:59.101386899Z No resources found in openshift-etcd namespace. 2026-11-04T04:10:59.243643622Z INFO: "metallb-operator" not detected. Skipping. 2026-11-04T04:10:59.299795594Z error: the server doesn't have a resource type "performanceprofile" 2026-11-04T04:10:59.461067694Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-11-04T04:10:59.461157961Z INFO: on-disk MachineConfig config collection complete. 2026-11-04T04:10:59.479426912Z INFO: Found 1 replicas - prometheus-k8s-0 2026-11-04T04:10:59.676373170Z INFO: "nfd" not detected. Skipping. 2026-11-04T04:10:59.866632780Z INFO: "sriov-network-operator" not detected. Skipping. 2026-11-04T04:10:59.930706423Z ERROR: No running kube-apiserver pods found 2026-11-04T04:10:59.944266840Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-8hg8l 2026-11-04T04:10:59.944266840Z insights-runtime-extractor-8dhxs 2026-11-04T04:10:59.944266840Z insights-runtime-extractor-hh9rs 2026-11-04T04:10:59.944266840Z insights-runtime-extractor-p7tlz 2026-11-04T04:10:59.966127613Z INFO: Worker host service log collection to complete. 2026-11-04T04:11:00.248696276Z INFO: Waiting for HAProxy config collection to complete ... 2026-11-04T04:11:00.253501484Z Wrote inspect data to must-gather. 2026-11-04T04:11:00.327522508Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-11-04T04:11:00.327522508Z See 'oc rsync -h' for help and examples 2026-11-04T04:11:00.338030013Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-11-04T04:11:00.400648580Z INFO: 'previous-logs' folder not found on ip-10-0-130-151.us-west-2.compute.internal, skipping... 2026-11-04T04:11:00.556331034Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-11-04T04:11:00.732442054Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-11-04T04:11:00.737003442Z Wrote inspect data to must-gather. 2026-11-04T04:11:00.906415265Z No resources found 2026-11-04T04:11:00.938867979Z Wrote inspect data to must-gather. 2026-11-04T04:11:01.131981903Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:01.189328282Z Wrote inspect data to must-gather. 2026-11-04T04:11:01.190840720Z Wrote inspect data to must-gather. 2026-11-04T04:11:01.234258799Z INFO: 'previous-logs' folder not found on ip-10-0-132-179.us-west-2.compute.internal, skipping... 2026-11-04T04:11:01.339685059Z error: the server doesn't have a resource type "machineconfigs" 2026-11-04T04:11:01.348104423Z tar: Removing leading `/' from member names 2026-11-04T04:11:01.455179083Z INFO: HAProxy config collection complete. 2026-11-04T04:11:01.466565702Z Wrote inspect data to must-gather. 2026-11-04T04:11:01.628466932Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-11-04T04:11:01.691968257Z INFO: 'previous-logs' folder not found on ip-10-0-137-184.us-west-2.compute.internal, skipping... 2026-11-04T04:11:01.692011575Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-11-04T04:11:01.692011575Z INFO: Machine Config Daemon termination log collection complete. 2026-11-04T04:11:01.725225981Z error: the server doesn't have a resource type "machineconfigpools" 2026-11-04T04:11:01.788210818Z Wrote inspect data to must-gather. 2026-11-04T04:11:01.861336104Z Wrote inspect data to must-gather. 2026-11-04T04:11:02.220039418Z INFO: Getting rules from prometheus-k8s-0 2026-11-04T04:11:02.303605351Z Wrote inspect data to must-gather. 2026-11-04T04:11:02.380489680Z Wrote inspect data to must-gather. 2026-11-04T04:11:02.669892685Z Gathering data for ns/openshift-cloud-credential-operator... 2026-11-04T04:11:02.768192397Z error: the server doesn't have a resource type "kubeletconfigs" 2026-11-04T04:11:02.873955144Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io 2026-11-04T04:11:03.069110683Z Wrote inspect data to must-gather. 2026-11-04T04:11:03.236102339Z Wrote inspect data to must-gather. 2026-11-04T04:11:03.407095693Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:03.487316485Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-11-04T04:11:03.661452413Z Wrote inspect data to must-gather. 2026-11-04T04:11:03.788489951Z INFO: LEGACY MODE 2026-11-04T04:11:03.788534852Z INFO: Gathering ovn-kubernetes master data 2026-11-04T04:11:03.788534852Z INFO: Getting status/config from prometheus-k8s-0 2026-11-04T04:11:04.073357134Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:04.075402435Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-11-04T04:11:04.076950212Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-11-04T04:11:04.250309633Z Wrote inspect data to must-gather. 2026-11-04T04:11:04.553327300Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:04.854179851Z Wrote inspect data to must-gather. 2026-11-04T04:11:05.119140441Z Gathering data for ns/openshift-etcd... 2026-11-04T04:11:05.211435585Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:05.216987971Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-11-04T04:11:05.342251973Z Wrote inspect data to must-gather. 2026-11-04T04:11:05.346031489Z INFO: Getting status/flags from prometheus-k8s-0 2026-11-04T04:11:05.706007558Z Gathering data for ns/openshift-monitoring... 2026-11-04T04:11:05.798701954Z Wrote inspect data to must-gather. 2026-11-04T04:11:05.819820631Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:05.859290115Z INFO: Waiting for network log collection to complete ... 2026-11-04T04:11:05.859341075Z INFO: Waiting for ovnk database copies to complete ... 2026-11-04T04:11:05.859845081Z INFO: Copying ovnk databases complete. 2026-11-04T04:11:05.861876076Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory 2026-11-04T04:11:05.862601169Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 783 is not a child of this shell 2026-11-04T04:11:05.862601169Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 784 is not a child of this shell 2026-11-04T04:11:05.862601169Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 785 is not a child of this shell 2026-11-04T04:11:05.862601169Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 786 is not a child of this shell 2026-11-04T04:11:05.862601169Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 787 is not a child of this shell 2026-11-04T04:11:05.862601169Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 789 is not a child of this shell 2026-11-04T04:11:05.862624982Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 790 is not a child of this shell 2026-11-04T04:11:05.862624982Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 791 is not a child of this shell 2026-11-04T04:11:05.862624982Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 793 is not a child of this shell 2026-11-04T04:11:05.862624982Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 794 is not a child of this shell 2026-11-04T04:11:05.862624982Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 796 is not a child of this shell 2026-11-04T04:11:05.862624982Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 798 is not a child of this shell 2026-11-04T04:11:05.862624982Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 799 is not a child of this shell 2026-11-04T04:11:05.866534510Z tar: ovnk_database_store: Cannot stat: No such file or directory 2026-11-04T04:11:05.870210753Z tar: Exiting with failure status due to previous errors 2026-11-04T04:11:05.870799600Z INFO: Network log collection complete. 2026-11-04T04:11:06.137075862Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:06.268917885Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-11-04T04:11:06.460417860Z Wrote inspect data to must-gather. 2026-11-04T04:11:06.472690560Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:06.475241012Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-11-04T04:11:06.883536391Z Wrote inspect data to must-gather. 2026-11-04T04:11:06.883579623Z error: inspection completed with the errors occurred while gathering data: 2026-11-04T04:11:06.883579623Z namespaces "assisted-installer" not found 2026-11-04T04:11:07.009765985Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:07.154894691Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-11-04T04:11:07.308586988Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod 2026-11-04T04:11:07.652501674Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:07.655568389Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-11-04T04:11:07.676375161Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c 2026-11-04T04:11:08.048208875Z daemonset.apps/perf-node-gather-daemonset created 2026-11-04T04:11:08.171938864Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:08.319775077Z Waiting for performance profile collector pods to become ready: 1 2026-11-04T04:11:08.379519990Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-11-04T04:11:08.811621036Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:08.814194301Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-11-04T04:11:09.215696508Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:09.584526310Z Waiting for performance profile collector pods to become ready: 2 2026-11-04T04:11:09.658558806Z INFO: Getting status from alertmanager-main-0 2026-11-04T04:11:09.860797434Z Wrote inspect data to /must-gather/istio. 2026-11-04T04:11:10.842529352Z Waiting for performance profile collector pods to become ready: 3 2026-11-04T04:11:11.141590805Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:11.760322248Z Done executing Istio gather script 2026-11-04T04:11:12.104817102Z Waiting for performance profile collector pods to become ready: 4 2026-11-04T04:11:13.361653708Z Waiting for performance profile collector pods to become ready: 5 2026-11-04T04:11:14.638609734Z Waiting for performance profile collector pods to become ready: 6 2026-11-04T04:11:15.892777689Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-11-04T04:11:16.146198799Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:16.470128196Z Collecting performance related data for node ip-10-0-137-184.us-west-2.compute.internal 2026-11-04T04:11:16.476669815Z Collecting performance related data for node ip-10-0-132-179.us-west-2.compute.internal 2026-11-04T04:11:16.483877917Z Collecting performance related data for node ip-10-0-130-151.us-west-2.compute.internal 2026-11-04T04:11:21.150653773Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:26.156672612Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:27.210763560Z Gathering data for ns/openshift-network-console... 2026-11-04T04:11:29.910765041Z Gathering data for ns/openshift-console-operator... 2026-11-04T04:11:30.650675111Z Collecting kubelet logs for node ip-10-0-137-184.us-west-2.compute.internal 2026-11-04T04:11:30.652377400Z Collecting kubelet logs for node ip-10-0-132-179.us-west-2.compute.internal 2026-11-04T04:11:30.654052421Z Collecting kubelet logs for node ip-10-0-130-151.us-west-2.compute.internal 2026-11-04T04:11:31.161380612Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:31.573120914Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-mnwsh namespace 2026-11-04T04:11:31.624374304Z INFO: Node performance data collection complete. 2026-11-04T04:11:32.224289069Z Gathering data for ns/openshift-console... 2026-11-04T04:11:35.413254135Z Gathering data for ns/openshift-cluster-storage-operator... 2026-11-04T04:11:36.166027276Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:38.585269145Z Gathering data for ns/openshift-dns-operator... 2026-11-04T04:11:40.694736503Z Gathering data for ns/openshift-dns... 2026-11-04T04:11:41.170546979Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:46.174957201Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:46.908354888Z Gathering data for ns/openshift-image-registry... 2026-11-04T04:11:50.779934095Z Gathering data for ns/openshift-ingress-operator... 2026-11-04T04:11:51.179496880Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:52.784689630Z Gathering data for ns/openshift-ingress... 2026-11-04T04:11:55.208589278Z Gathering data for ns/openshift-ingress-canary... 2026-11-04T04:11:56.183978919Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:11:58.527438833Z Gathering data for ns/openshift-insights... 2026-11-04T04:12:01.188442988Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:06.193294004Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:07.099797733Z Gathering data for ns/openshift-kueue-operator... 2026-11-04T04:12:11.197708510Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:11.414365835Z Gathering data for ns/openshift-jobset-operator... 2026-11-04T04:12:14.893690111Z Gathering data for ns/openshift-config... 2026-11-04T04:12:16.202052973Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:16.776945533Z Gathering data for ns/openshift-config-managed... 2026-11-04T04:12:18.828465952Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-11-04T04:12:20.723707550Z Gathering data for ns/openshift-kube-apiserver... 2026-11-04T04:12:21.206531040Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:22.616705522Z Gathering data for ns/openshift-kube-controller-manager... 2026-11-04T04:12:24.547133233Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-11-04T04:12:26.211214472Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:26.578106609Z Gathering data for ns/openshift-kube-scheduler... 2026-11-04T04:12:28.461792133Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-11-04T04:12:30.471758249Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-11-04T04:12:31.215793586Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:33.252361693Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-11-04T04:12:35.635653229Z Gathering data for ns/openshift-user-workload-monitoring... 2026-11-04T04:12:36.220295526Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:38.404148526Z Gathering data for ns/openshift-multus... 2026-11-04T04:12:41.224773299Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:46.229246797Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:51.233669880Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:12:55.476777410Z Gathering data for ns/openshift-ovn-kubernetes... 2026-11-04T04:12:56.238142125Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:01.242582209Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:06.247117038Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:09.327790009Z Gathering data for ns/openshift-host-network... 2026-11-04T04:13:11.251462955Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:11.477224386Z Gathering data for ns/openshift-network-diagnostics... 2026-11-04T04:13:16.176229753Z Gathering data for ns/openshift-network-node-identity... 2026-11-04T04:13:16.255940225Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:19.281896988Z Gathering data for ns/openshift-network-operator... 2026-11-04T04:13:21.260547664Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:22.514801239Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-11-04T04:13:24.353664121Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-11-04T04:13:26.265026105Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:27.977131639Z Gathering data for ns/openshift-apiserver-operator... 2026-11-04T04:13:29.848273838Z Gathering data for ns/openshift-apiserver... 2026-11-04T04:13:31.270532332Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:32.417827059Z Gathering data for ns/openshift-controller-manager-operator... 2026-11-04T04:13:34.258413733Z Gathering data for ns/openshift-controller-manager... 2026-11-04T04:13:36.169947970Z Gathering data for ns/openshift-cluster-samples-operator... 2026-11-04T04:13:36.275313567Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:39.405081148Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-11-04T04:13:41.279936537Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:41.335666646Z Gathering data for ns/openshift-service-ca-operator... 2026-11-04T04:13:43.633152435Z Gathering data for ns/openshift-service-ca... 2026-11-04T04:13:46.284748557Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:46.285954134Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-11-04T04:13:51.289252582Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70 2026-11-04T04:13:52.526598519Z Wrote inspect data to must-gather. 2026-11-04T04:13:52.526645283Z error: inspection completed with the errors occurred while gathering data: 2026-11-04T04:13:52.526645283Z skipping gathering secrets/support due to error: secrets "support" not found 2026-11-04T04:13:53.001569173Z error: the server doesn't have a resource type "clusters" 2026-11-04T04:13:53.040430094Z Caches written to disk