2026-10-31T01:30:23.544900856Z [disk usage checker] Started 2026-10-31T01:30:23.548253708Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:23.879221390Z Gathering data for ns/openshift-cluster-version... 2026-10-31T01:30:24.059158545Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-10-31T01:30:24.187459446Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T01:30:24.307313834Z Gathering data for ns/default... 2026-10-31T01:30:24.719890763Z Gathering data for ns/openshift... 2026-10-31T01:30:25.236424146Z Gathering data for ns/kube-system... 2026-10-31T01:30:26.509934044Z Gathering data for ns/openshift-etcd... 2026-10-31T01:30:26.589403290Z Waiting on subprocesses to finish execution. 2026-10-31T01:30:26.601148307Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. 2026-10-31T01:30:26.601905008Z INFO: Gathering HAProxy config files 2026-10-31T01:30:26.611597428Z INFO: Collecting host service logs for crio 2026-10-31T01:30:26.611912574Z INFO: Collecting host service logs for kubelet 2026-10-31T01:30:26.612205854Z INFO: Collecting host service logs for rpm-ostreed 2026-10-31T01:30:26.612487261Z INFO: Collecting host service logs for ostree-finalize-staged 2026-10-31T01:30:26.612800460Z INFO: Collecting host service logs for machine-config-daemon-firstboot 2026-10-31T01:30:26.613084244Z INFO: Collecting host service logs for machine-config-daemon-host 2026-10-31T01:30:26.613399028Z INFO: Collecting host service logs for NetworkManager 2026-10-31T01:30:26.613783898Z INFO: Collecting host service logs for openvswitch 2026-10-31T01:30:26.614114365Z INFO: Collecting host service logs for ovs-configuration 2026-10-31T01:30:26.614375129Z INFO: Collecting host service logs for ovsdb-server 2026-10-31T01:30:26.614669851Z INFO: Collecting host service logs for ovs-vswitchd 2026-10-31T01:30:26.618827499Z INFO: Gathering on-disk MachineConfig from degraded nodes 2026-10-31T01:30:26.618963513Z INFO: Waiting for worker host service log collection to complete ... 2026-10-31T01:30:26.622538993Z Executing Istio gather script 2026-10-31T01:30:26.626628477Z INFO: Gathering machine config daemon's old logs from all nodes 2026-10-31T01:30:26.835454441Z INFO: Waiting for node performance related collection to complete ... 2026-10-31T01:30:27.254662487Z No resources found in openshift-etcd namespace. 2026-10-31T01:30:27.336647066Z INFO: "nfd" not detected. Skipping. 2026-10-31T01:30:27.399393233Z No resources found 2026-10-31T01:30:27.528567767Z INFO: Waiting for on-disk MachineConfig collection to complete ... 2026-10-31T01:30:27.528612425Z INFO: on-disk MachineConfig config collection complete. 2026-10-31T01:30:27.813504621Z Wrote inspect data to must-gather. 2026-10-31T01:30:27.813552565Z error: inspection completed with the errors occurred while gathering data: 2026-10-31T01:30:27.813552565Z namespaces "assisted-installer" not found 2026-10-31T01:30:28.031600167Z INFO: Found 1 replicas - prometheus-k8s-0 2026-10-31T01:30:28.055626973Z INFO: "metallb-operator" not detected. Skipping. 2026-10-31T01:30:28.418526480Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-bgqsr 2026-10-31T01:30:28.418526480Z insights-runtime-extractor-4f7jl 2026-10-31T01:30:28.418526480Z insights-runtime-extractor-gnddf 2026-10-31T01:30:28.418526480Z insights-runtime-extractor-w49pc 2026-10-31T01:30:28.467314285Z error: the server doesn't have a resource type "multi-networkpolicy" 2026-10-31T01:30:28.530493030Z Wrote inspect data to must-gather. 2026-10-31T01:30:28.626887504Z INFO: "sriov-network-operator" not detected. Skipping. 2026-10-31T01:30:28.773224784Z ERROR: No running kube-apiserver pods found 2026-10-31T01:30:28.780764440Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:28.783566262Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. 2026-10-31T01:30:28.904268621Z No resources found 2026-10-31T01:30:28.923694096Z INFO: Worker host service log collection to complete. 2026-10-31T01:30:28.975514377Z INFO: namespace openshift-frr-k8s not detected. Skipping. 2026-10-31T01:30:29.012642213Z Wrote inspect data to must-gather. 2026-10-31T01:30:29.028392523Z Wrote inspect data to must-gather. 2026-10-31T01:30:29.108015032Z error: the server doesn't have a resource type "performanceprofile" 2026-10-31T01:30:29.141589316Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments 2026-10-31T01:30:29.141589316Z See 'oc rsync -h' for help and examples 2026-10-31T01:30:29.199015015Z Wrote inspect data to must-gather. 2026-10-31T01:30:29.209498713Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T01:30:29.324729158Z Wrote inspect data to must-gather. 2026-10-31T01:30:29.372550541Z INFO: 'previous-logs' folder not found on ip-10-0-129-191.us-east-2.compute.internal, skipping... 2026-10-31T01:30:29.387163844Z INFO: Waiting for HAProxy config collection to complete ... 2026-10-31T01:30:29.406084224Z INFO: Getting alertmanagers from prometheus-k8s-0 2026-10-31T01:30:29.470733607Z Wrote inspect data to must-gather. 2026-10-31T01:30:29.678148923Z Wrote inspect data to must-gather. 2026-10-31T01:30:29.682505565Z Wrote inspect data to must-gather. 2026-10-31T01:30:29.722840728Z tar: Removing leading `/' from member names 2026-10-31T01:30:29.732752360Z INFO: HAProxy config collection complete. 2026-10-31T01:30:29.791496393Z INFO: 'previous-logs' folder not found on ip-10-0-135-193.us-east-2.compute.internal, skipping... 2026-10-31T01:30:29.918208784Z error: the server doesn't have a resource type "machineconfigs" 2026-10-31T01:30:29.950398615Z INFO: Getting rules from prometheus-k8s-0 2026-10-31T01:30:29.968835483Z Gathering data for ns/openshift-cloud-credential-operator... 2026-10-31T01:30:30.049058712Z Wrote inspect data to must-gather. 2026-10-31T01:30:30.181243117Z error: the server doesn't have a resource type "machineconfigpools" 2026-10-31T01:30:30.193873374Z INFO: 'previous-logs' folder not found on ip-10-0-143-213.us-east-2.compute.internal, skipping... 2026-10-31T01:30:30.193919909Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... 2026-10-31T01:30:30.193945701Z INFO: Machine Config Daemon termination log collection complete. 2026-10-31T01:30:30.254663179Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+ 2026-10-31T01:30:30.347587837Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. 2026-10-31T01:30:30.435211187Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2026-10-31T01:30:30.447002635Z Wrote inspect data to must-gather. 2026-10-31T01:30:30.469661506Z INFO: Getting status/config from prometheus-k8s-0 2026-10-31T01:30:30.499643085Z Wrote inspect data to must-gather. 2026-10-31T01:30:30.712472826Z error: the server doesn't have a resource type "kubeletconfigs" 2026-10-31T01:30:30.818102123Z INFO: LEGACY MODE 2026-10-31T01:30:30.818137956Z INFO: Gathering ovn-kubernetes master data 2026-10-31T01:30:30.966028416Z INFO: Getting status/flags from prometheus-k8s-0 2026-10-31T01:30:31.005121734Z Wrote inspect data to must-gather. 2026-10-31T01:30:31.010016631Z Gathering data for ns/openshift-monitoring... 2026-10-31T01:30:31.232042107Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 2026-10-31T01:30:31.252895087Z Wrote inspect data to must-gather. 2026-10-31T01:30:31.481390970Z INFO: Getting targets?state=active from prometheus-k8s-0 2026-10-31T01:30:31.511869730Z Wrote inspect data to must-gather. 2026-10-31T01:30:31.723756101Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io 2026-10-31T01:30:31.734328993Z INFO: Getting status/tsdb from prometheus-k8s-0 2026-10-31T01:30:31.761624001Z INFO: Waiting for network log collection to complete ... 2026-10-31T01:30:31.761663225Z INFO: Waiting for ovnk database copies to complete ... 2026-10-31T01:30:31.762120782Z INFO: Copying ovnk databases complete. 2026-10-31T01:30:31.763881532Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory 2026-10-31T01:30:31.764302875Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 676 is not a child of this shell 2026-10-31T01:30:31.764302875Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 677 is not a child of this shell 2026-10-31T01:30:31.764302875Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 678 is not a child of this shell 2026-10-31T01:30:31.764302875Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 679 is not a child of this shell 2026-10-31T01:30:31.764302875Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 680 is not a child of this shell 2026-10-31T01:30:31.764319180Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 681 is not a child of this shell 2026-10-31T01:30:31.764319180Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 682 is not a child of this shell 2026-10-31T01:30:31.764319180Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 683 is not a child of this shell 2026-10-31T01:30:31.764319180Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 698 is not a child of this shell 2026-10-31T01:30:31.764319180Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 699 is not a child of this shell 2026-10-31T01:30:31.764319180Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 700 is not a child of this shell 2026-10-31T01:30:31.764319180Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 701 is not a child of this shell 2026-10-31T01:30:31.764330716Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 702 is not a child of this shell 2026-10-31T01:30:31.767700703Z tar: ovnk_database_store: Cannot stat: No such file or directory 2026-10-31T01:30:31.770875138Z Wrote inspect data to must-gather. 2026-10-31T01:30:31.772596556Z tar: Exiting with failure status due to previous errors 2026-10-31T01:30:31.773615497Z INFO: Network log collection complete. 2026-10-31T01:30:31.941427026Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:32.017919835Z Wrote inspect data to must-gather. 2026-10-31T01:30:32.118051042Z INFO: Getting status from alertmanager-main-0 2026-10-31T01:30:32.173704054Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:32.177423589Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io 2026-10-31T01:30:32.264295675Z Wrote inspect data to must-gather. 2026-10-31T01:30:32.377944796Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:32.486532333Z Wrote inspect data to must-gather. 2026-10-31T01:30:32.592785386Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:32.596452458Z Inspecting resource crd/gateways.gateway.networking.k8s.io 2026-10-31T01:30:32.790768816Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:32.792818064Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod 2026-10-31T01:30:32.942264330Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c 2026-10-31T01:30:33.019246350Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:33.023043852Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io 2026-10-31T01:30:33.095092420Z daemonset.apps/perf-node-gather-daemonset created 2026-10-31T01:30:33.249953430Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:33.258043071Z Waiting for performance profile collector pods to become ready: 1 2026-10-31T01:30:33.453698640Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:33.457050405Z Inspecting resource crd/httproutes.gateway.networking.k8s.io 2026-10-31T01:30:33.722747843Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:33.792149097Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:33.919925955Z Gathering data for ns/openshift-network-console... 2026-10-31T01:30:33.944395159Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:33.947479548Z Inspecting resource crd/inferencepools.inference.networking.k8s.io 2026-10-31T01:30:34.132093409Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:34.359127158Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:34.362953202Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io 2026-10-31T01:30:34.423733663Z Waiting for performance profile collector pods to become ready: 2 2026-10-31T01:30:34.509904742Z Gathering data for ns/openshift-console-operator... 2026-10-31T01:30:34.552494195Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:34.758288543Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:34.762424114Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io 2026-10-31T01:30:34.949008602Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:35.005306226Z Gathering data for ns/openshift-console... 2026-10-31T01:30:35.170044557Z Wrote inspect data to /must-gather/istio. 2026-10-31T01:30:35.585902514Z Waiting for performance profile collector pods to become ready: 3 2026-10-31T01:30:35.710281397Z Gathering data for ns/openshift-cluster-storage-operator... 2026-10-31T01:30:36.083156928Z Done executing Istio gather script 2026-10-31T01:30:36.352405021Z Gathering data for ns/openshift-dns-operator... 2026-10-31T01:30:36.728616319Z Waiting for performance profile collector pods to become ready: 4 2026-10-31T01:30:36.778801287Z Gathering data for ns/openshift-dns... 2026-10-31T01:30:37.604832019Z Gathering data for ns/openshift-image-registry... 2026-10-31T01:30:37.863143855Z Waiting for performance profile collector pods to become ready: 5 2026-10-31T01:30:38.253348237Z Gathering data for ns/openshift-ingress-operator... 2026-10-31T01:30:38.688982313Z Gathering data for ns/openshift-ingress... 2026-10-31T01:30:38.797752882Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:38.988959615Z Waiting for performance profile collector pods to become ready: 6 2026-10-31T01:30:39.133814732Z Gathering data for ns/openshift-ingress-canary... 2026-10-31T01:30:39.662472376Z Gathering data for ns/openshift-insights... 2026-10-31T01:30:40.123745325Z Waiting for performance profile collector pods to become ready: 7 2026-10-31T01:30:41.245468116Z Daemonset perf-node-gather-daemonset ready 3 out of 3 2026-10-31T01:30:41.499944714Z Collecting performance related data for node ip-10-0-135-193.us-east-2.compute.internal 2026-10-31T01:30:41.506918721Z Collecting performance related data for node ip-10-0-129-191.us-east-2.compute.internal 2026-10-31T01:30:41.514661347Z Collecting performance related data for node ip-10-0-143-213.us-east-2.compute.internal 2026-10-31T01:30:42.855789616Z Gathering data for ns/kserve... 2026-10-31T01:30:43.803888707Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:45.945486709Z Gathering data for ns/openshift-config... 2026-10-31T01:30:46.313397750Z Gathering data for ns/openshift-config-managed... 2026-10-31T01:30:46.799407308Z Gathering data for ns/openshift-kube-apiserver-operator... 2026-10-31T01:30:47.208239295Z Gathering data for ns/openshift-kube-apiserver... 2026-10-31T01:30:47.574149130Z Gathering data for ns/openshift-kube-controller-manager... 2026-10-31T01:30:47.956428807Z Gathering data for ns/openshift-kube-controller-manager-operator... 2026-10-31T01:30:48.393183215Z Gathering data for ns/openshift-kube-scheduler... 2026-10-31T01:30:48.809943113Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:48.821731479Z Gathering data for ns/openshift-kube-scheduler-operator... 2026-10-31T01:30:48.864081026Z Collecting kubelet logs for node ip-10-0-135-193.us-east-2.compute.internal 2026-10-31T01:30:48.865986810Z Collecting kubelet logs for node ip-10-0-129-191.us-east-2.compute.internal 2026-10-31T01:30:48.868142286Z Collecting kubelet logs for node ip-10-0-143-213.us-east-2.compute.internal 2026-10-31T01:30:49.288493941Z Gathering data for ns/openshift-kube-storage-version-migrator... 2026-10-31T01:30:49.401889052Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-whtv4 namespace 2026-10-31T01:30:49.411061533Z INFO: Node performance data collection complete. 2026-10-31T01:30:49.792823042Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... 2026-10-31T01:30:50.222249051Z Gathering data for ns/openshift-user-workload-monitoring... 2026-10-31T01:30:50.864710138Z Gathering data for ns/openshift-multus... 2026-10-31T01:30:52.818721637Z Gathering data for ns/openshift-ovn-kubernetes... 2026-10-31T01:30:53.815871953Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:54.619803615Z Gathering data for ns/openshift-host-network... 2026-10-31T01:30:55.037829675Z Gathering data for ns/openshift-network-diagnostics... 2026-10-31T01:30:55.767477692Z Gathering data for ns/openshift-network-node-identity... 2026-10-31T01:30:56.342218755Z Gathering data for ns/openshift-network-operator... 2026-10-31T01:30:56.851291675Z Gathering data for ns/openshift-cloud-network-config-controller... 2026-10-31T01:30:57.211481165Z Gathering data for ns/openshift-cluster-node-tuning-operator... 2026-10-31T01:30:57.850081664Z Gathering data for ns/openshift-apiserver-operator... 2026-10-31T01:30:58.235962307Z Gathering data for ns/openshift-apiserver... 2026-10-31T01:30:58.752141698Z Gathering data for ns/openshift-controller-manager-operator... 2026-10-31T01:30:58.820910662Z [disk usage checker] Volume usage percentage: current = 36 ; allowed = 70 2026-10-31T01:30:59.094802927Z Gathering data for ns/openshift-controller-manager... 2026-10-31T01:30:59.501397718Z Gathering data for ns/openshift-cluster-samples-operator... 2026-10-31T01:31:00.290526312Z Gathering data for ns/openshift-operator-lifecycle-manager... 2026-10-31T01:31:00.666651738Z Gathering data for ns/openshift-service-ca-operator... 2026-10-31T01:31:01.076405672Z Gathering data for ns/openshift-service-ca... 2026-10-31T01:31:01.562781916Z Gathering data for ns/openshift-cluster-csi-drivers... 2026-10-31T01:31:02.696496531Z Wrote inspect data to must-gather. 2026-10-31T01:31:02.696540619Z error: inspection completed with the errors occurred while gathering data: 2026-10-31T01:31:02.696540619Z skipping gathering secrets/support due to error: secrets "support" not found 2026-10-31T01:31:02.837479452Z error: the server doesn't have a resource type "clusters" 2026-10-31T01:31:03.000489713Z Caches written to disk