2026-10-31T04:21:18.621920906Z [disk usage checker] Started 2026-10-31T04:21:18.625089421Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:19.470370782Z Gathering data for ns/openshift-cluster-version... 2026-10-31T04:21:20.166460037Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-10-31T04:21:20.663717183Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T04:21:21.118143185Z Gathering data for ns/default... 2026-10-31T04:21:22.807641957Z Gathering data for ns/openshift... 2026-10-31T04:21:23.629464996Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:24.585496847Z Gathering data for ns/kube-system... 2026-10-31T04:21:25.450056974Z Waiting on subprocesses to finish execution. 2026-10-31T04:21:25.456957950Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-10-31T04:21:25.458506429Z INFO: Gathering machine config daemon's old logs from all nodes 2026-10-31T04:21:25.459264839Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-10-31T04:21:25.481842312Z Executing Istio gather script 2026-10-31T04:21:25.489254596Z INFO: Gathering HAProxy config files 2026-10-31T04:21:25.499516651Z INFO: Collecting host service logs for crio 2026-10-31T04:21:25.499842527Z INFO: Collecting host service logs for kubelet 2026-10-31T04:21:25.500201301Z INFO: Collecting host service logs for rpm-ostreed 2026-10-31T04:21:25.500497059Z INFO: Collecting host service logs for ostree-finalize-staged 2026-10-31T04:21:25.500814858Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-10-31T04:21:25.501255517Z INFO: Collecting host service logs for machine-config-daemon-host 2026-10-31T04:21:25.501594089Z INFO: Collecting host service logs for NetworkManager 2026-10-31T04:21:25.501931691Z INFO: Collecting host service logs for openvswitch 2026-10-31T04:21:25.502232084Z INFO: Collecting host service logs for ovs-configuration 2026-10-31T04:21:25.502537300Z INFO: Collecting host service logs for ovsdb-server 2026-10-31T04:21:25.502846428Z INFO: Collecting host service logs for ovs-vswitchd 2026-10-31T04:21:25.503392923Z INFO: Waiting for worker host service log collection to complete ... 2026-10-31T04:21:25.510210666Z INFO: Waiting for node performance related collection to complete ... 2026-10-31T04:21:26.007351609Z No resources found in openshift-etcd namespace. 2026-10-31T04:21:26.159357200Z INFO: "metallb-operator" not detected. Skipping. 2026-10-31T04:21:26.186722460Z INFO: "nfd" not detected. Skipping. 2026-10-31T04:21:26.545143221Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-8fnbn 2026-10-31T04:21:26.550282286Z insights-runtime-extractor-4sthp 2026-10-31T04:21:26.550410050Z insights-runtime-extractor-9gnwr 2026-10-31T04:21:26.550451016Z insights-runtime-extractor-lc845 2026-10-31T04:21:26.591375639Z error: the server doesn't have a resource type "performanceprofile" 2026-10-31T04:21:26.600192596Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-10-31T04:21:26.784942962Z INFO: Found 1 replicas - prometheus-k8s-0 2026-10-31T04:21:26.806010920Z No resources found 2026-10-31T04:21:26.844314760Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-10-31T04:21:26.844314760Z See 'oc rsync -h' for help and examples 2026-10-31T04:21:26.867400344Z INFO: "sriov-network-operator" not detected. Skipping. 2026-10-31T04:21:26.996967594Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-10-31T04:21:27.001921256Z INFO: on-disk MachineConfig config collection complete. 2026-10-31T04:21:27.014075635Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-10-31T04:21:27.121251069Z INFO: Worker host service log collection to complete. 2026-10-31T04:21:27.368828670Z Wrote inspect data to must-gather. 2026-10-31T04:21:27.569579622Z No resources found 2026-10-31T04:21:27.645769636Z INFO: Waiting for HAProxy config collection to complete ... 2026-10-31T04:21:27.700132312Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T04:21:27.713658789Z Wrote inspect data to must-gather. 2026-10-31T04:21:27.746987343Z INFO: 'previous-logs' folder not found on ip-10-0-132-152.us-west-2.compute.internal, skipping... 2026-10-31T04:21:27.800631197Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-10-31T04:21:27.851909826Z Wrote inspect data to must-gather. 2026-10-31T04:21:27.876554652Z ERROR: No running kube-apiserver pods found 2026-10-31T04:21:28.023773291Z Wrote inspect data to must-gather. 2026-10-31T04:21:28.038988216Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-10-31T04:21:28.077856580Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-10-31T04:21:28.096841722Z Wrote inspect data to must-gather. 2026-10-31T04:21:28.137383691Z Wrote inspect data to must-gather. 2026-10-31T04:21:28.165313800Z error: the server doesn't have a resource type "machineconfigs" 2026-10-31T04:21:28.251814848Z INFO: 'previous-logs' folder not found on ip-10-0-138-202.us-west-2.compute.internal, skipping... 2026-10-31T04:21:28.307712028Z Wrote inspect data to must-gather. 2026-10-31T04:21:28.594431614Z INFO: 'previous-logs' folder not found on ip-10-0-140-11.us-west-2.compute.internal, skipping... 2026-10-31T04:21:28.594478728Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-10-31T04:21:28.594478728Z INFO: Machine Config Daemon termination log collection complete. 2026-10-31T04:21:28.610481201Z tar: Removing leading `/' from member names 2026-10-31T04:21:28.612711186Z error: the server doesn't have a resource type "machineconfigpools" 2026-10-31T04:21:28.633877885Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:28.661104247Z Wrote inspect data to must-gather. 2026-10-31T04:21:28.661583395Z INFO: HAProxy config collection complete. 2026-10-31T04:21:29.041515738Z Wrote inspect data to must-gather. 2026-10-31T04:21:29.273861467Z Wrote inspect data to must-gather. 2026-10-31T04:21:29.382328812Z INFO: Getting rules from prometheus-k8s-0 2026-10-31T04:21:29.476491147Z error: the server doesn't have a resource type "kubeletconfigs" 2026-10-31T04:21:29.508972287Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io 2026-10-31T04:21:29.922144035Z Wrote inspect data to must-gather. 2026-10-31T04:21:29.954912904Z Wrote inspect data to must-gather. 2026-10-31T04:21:29.992408979Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:30.148502419Z Gathering data for ns/openshift-cloud-credential-operator... 2026-10-31T04:21:30.374708333Z Wrote inspect data to must-gather. 2026-10-31T04:21:30.616005426Z INFO: LEGACY MODE 2026-10-31T04:21:30.616051038Z INFO: Gathering ovn-kubernetes master data 2026-10-31T04:21:30.638150669Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:30.642032914Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-10-31T04:21:30.802457678Z Wrote inspect data to must-gather. 2026-10-31T04:21:30.832273737Z INFO: Getting status/config from prometheus-k8s-0 2026-10-31T04:21:30.855816995Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-10-31T04:21:31.076136884Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:31.192544559Z Wrote inspect data to must-gather. 2026-10-31T04:21:31.398832325Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T04:21:31.663997773Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:31.667806831Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-10-31T04:21:31.680170618Z Wrote inspect data to must-gather. 2026-10-31T04:21:32.212979396Z Wrote inspect data to must-gather. 2026-10-31T04:21:32.227367238Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:32.338191352Z Gathering data for ns/openshift-etcd... 2026-10-31T04:21:32.349236828Z INFO: Getting status/flags from prometheus-k8s-0 2026-10-31T04:21:32.646329068Z INFO: Waiting for network log collection to complete ... 2026-10-31T04:21:32.646386592Z INFO: Waiting for ovnk database copies to complete ... 2026-10-31T04:21:32.646812628Z INFO: Copying ovnk databases complete. 2026-10-31T04:21:32.648626141Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory 2026-10-31T04:21:32.649101398Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 729 is not a child of this shell 2026-10-31T04:21:32.649101398Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 730 is not a child of this shell 2026-10-31T04:21:32.649101398Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 731 is not a child of this shell 2026-10-31T04:21:32.649101398Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 732 is not a child of this shell 2026-10-31T04:21:32.649101398Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 733 is not a child of this shell 2026-10-31T04:21:32.649101398Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 734 is not a child of this shell 2026-10-31T04:21:32.649133569Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 735 is not a child of this shell 2026-10-31T04:21:32.649133569Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 736 is not a child of this shell 2026-10-31T04:21:32.649133569Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 738 is not a child of this shell 2026-10-31T04:21:32.649133569Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 739 is not a child of this shell 2026-10-31T04:21:32.649133569Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 740 is not a child of this shell 2026-10-31T04:21:32.649133569Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 741 is not a child of this shell 2026-10-31T04:21:32.649133569Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 742 is not a child of this shell 2026-10-31T04:21:32.652767948Z tar: ovnk_database_store: Cannot stat: No such file or directory 2026-10-31T04:21:32.658263820Z tar: Exiting with failure status due to previous errors 2026-10-31T04:21:32.659083027Z INFO: Network log collection complete. 2026-10-31T04:21:32.693655905Z Wrote inspect data to must-gather. 2026-10-31T04:21:32.849183935Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:32.851565173Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-10-31T04:21:32.858241838Z Gathering data for ns/openshift-monitoring... 2026-10-31T04:21:33.205077283Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-10-31T04:21:33.331871387Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:33.489569320Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod 2026-10-31T04:21:33.638271152Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:33.822413931Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c 2026-10-31T04:21:33.954745406Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:33.957527736Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-10-31T04:21:34.055552147Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-10-31T04:21:34.193405743Z daemonset.apps/perf-node-gather-daemonset created 2026-10-31T04:21:34.339803674Z Wrote inspect data to must-gather. 2026-10-31T04:21:34.339848100Z error: inspection completed with the errors occurred while gathering data: 2026-10-31T04:21:34.339848100Z namespaces "assisted-installer" not found 2026-10-31T04:21:34.448553721Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:34.454372038Z Waiting for performance profile collector pods to become ready: 1 2026-10-31T04:21:35.061096774Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:35.063761225Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-10-31T04:21:35.110011034Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-10-31T04:21:35.500152212Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:35.702953724Z Waiting for performance profile collector pods to become ready: 2 2026-10-31T04:21:36.090660428Z Wrote inspect data to /must-gather/istio. 2026-10-31T04:21:36.294995033Z INFO: Getting status from alertmanager-main-0 2026-10-31T04:21:36.953754565Z Waiting for performance profile collector pods to become ready: 3 2026-10-31T04:21:37.780918353Z Done executing Istio gather script 2026-10-31T04:21:38.205215575Z Waiting for performance profile collector pods to become ready: 4 2026-10-31T04:21:38.642734517Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:39.444975609Z Waiting for performance profile collector pods to become ready: 5 2026-10-31T04:21:40.702410316Z Waiting for performance profile collector pods to become ready: 6 2026-10-31T04:21:41.941692022Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-10-31T04:21:42.473899825Z Collecting performance related data for node ip-10-0-132-152.us-west-2.compute.internal 2026-10-31T04:21:42.480313371Z Collecting performance related data for node ip-10-0-140-11.us-west-2.compute.internal 2026-10-31T04:21:42.487222719Z Collecting performance related data for node ip-10-0-138-202.us-west-2.compute.internal 2026-10-31T04:21:43.647109268Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:48.651639918Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:52.601919827Z Gathering data for ns/openshift-network-console... 2026-10-31T04:21:53.656090282Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:21:55.051571700Z Gathering data for ns/openshift-console-operator... 2026-10-31T04:21:55.909073011Z Collecting kubelet logs for node ip-10-0-132-152.us-west-2.compute.internal 2026-10-31T04:21:55.910796619Z Collecting kubelet logs for node ip-10-0-140-11.us-west-2.compute.internal 2026-10-31T04:21:55.912491355Z Collecting kubelet logs for node ip-10-0-138-202.us-west-2.compute.internal 2026-10-31T04:21:56.767912152Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-pnctt namespace 2026-10-31T04:21:56.824971223Z INFO: Node performance data collection complete. 2026-10-31T04:21:57.239994155Z Gathering data for ns/openshift-console... 2026-10-31T04:21:58.660461981Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:00.136449665Z Gathering data for ns/openshift-cluster-storage-operator... 2026-10-31T04:22:03.056253185Z Gathering data for ns/openshift-dns-operator... 2026-10-31T04:22:03.664807406Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:04.965875416Z Gathering data for ns/openshift-dns... 2026-10-31T04:22:08.670402874Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:10.610322263Z Gathering data for ns/openshift-image-registry... 2026-10-31T04:22:13.674729042Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:14.042056850Z Gathering data for ns/openshift-ingress-operator... 2026-10-31T04:22:15.882407658Z Gathering data for ns/openshift-ingress... 2026-10-31T04:22:18.094768329Z Gathering data for ns/openshift-ingress-canary... 2026-10-31T04:22:18.679101476Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:21.103724007Z Gathering data for ns/openshift-insights... 2026-10-31T04:22:23.683375563Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:28.689347487Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:29.801315750Z Gathering data for ns/openshift-config... 2026-10-31T04:22:31.479451312Z Gathering data for ns/openshift-config-managed... 2026-10-31T04:22:33.324681582Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-10-31T04:22:33.693844933Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:35.002715397Z Gathering data for ns/openshift-kube-apiserver... 2026-10-31T04:22:36.686843621Z Gathering data for ns/openshift-kube-controller-manager... 2026-10-31T04:22:38.396740151Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-10-31T04:22:38.698208067Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:40.186264359Z Gathering data for ns/openshift-kube-scheduler... 2026-10-31T04:22:41.868982912Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-10-31T04:22:43.667085991Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-10-31T04:22:43.702487626Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:46.166954201Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-10-31T04:22:48.233929538Z Gathering data for ns/openshift-user-workload-monitoring... 2026-10-31T04:22:48.706965663Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:50.740361169Z Gathering data for ns/openshift-multus... 2026-10-31T04:22:53.711431339Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:22:58.715800253Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:03.720139563Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:06.271961514Z Gathering data for ns/openshift-ovn-kubernetes... 2026-10-31T04:23:08.724447649Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:13.728705625Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:18.733072169Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:18.930554876Z Gathering data for ns/openshift-host-network... 2026-10-31T04:23:20.889710172Z Gathering data for ns/openshift-network-diagnostics... 2026-10-31T04:23:23.737293498Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:25.165645653Z Gathering data for ns/openshift-network-node-identity... 2026-10-31T04:23:27.981843594Z Gathering data for ns/openshift-network-operator... 2026-10-31T04:23:28.741561300Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:30.918117120Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-10-31T04:23:32.611612117Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-10-31T04:23:33.745934487Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:35.931519243Z Gathering data for ns/openshift-apiserver-operator... 2026-10-31T04:23:37.666395695Z Gathering data for ns/openshift-apiserver... 2026-10-31T04:23:38.750786685Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:40.024977116Z Gathering data for ns/openshift-controller-manager-operator... 2026-10-31T04:23:41.709434470Z Gathering data for ns/openshift-controller-manager... 2026-10-31T04:23:43.447262564Z Gathering data for ns/openshift-cluster-samples-operator... 2026-10-31T04:23:43.755219699Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:46.380283370Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-10-31T04:23:48.135606203Z Gathering data for ns/openshift-service-ca-operator... 2026-10-31T04:23:48.759615037Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:50.237536644Z Gathering data for ns/openshift-service-ca... 2026-10-31T04:23:52.663326974Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-10-31T04:23:53.763965836Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:58.346530339Z Wrote inspect data to must-gather. 2026-10-31T04:23:58.346568113Z error: inspection completed with the errors occurred while gathering data: 2026-10-31T04:23:58.346568113Z skipping gathering secrets/support due to error: secrets "support" not found 2026-10-31T04:23:58.748407396Z error: the server doesn't have a resource type "clusters" 2026-10-31T04:23:58.768360152Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70 2026-10-31T04:23:58.800843096Z Caches written to disk