2026-11-03T20:18:02.073317881Z [disk usage checker] Started
2026-11-03T20:18:02.076766695Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:02.969602100Z Gathering data for ns/openshift-cluster-version...
2026-11-03T20:18:03.852365946Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T20:18:04.402729972Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T20:18:04.943500658Z Gathering data for ns/default...
2026-11-03T20:18:06.742551934Z Gathering data for ns/openshift...
2026-11-03T20:18:07.081167794Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:08.652256127Z Gathering data for ns/kube-system...
2026-11-03T20:18:10.659757603Z Waiting on subprocesses to finish execution.
2026-11-03T20:18:10.668123678Z INFO: Gathering HAProxy config files
2026-11-03T20:18:10.680871815Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-03T20:18:10.682834740Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-03T20:18:10.685443836Z INFO: Collecting host service logs for crio
2026-11-03T20:18:10.685741164Z INFO: Collecting host service logs for kubelet
2026-11-03T20:18:10.686027928Z INFO: Collecting host service logs for rpm-ostreed
2026-11-03T20:18:10.686355794Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-03T20:18:10.686666452Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-03T20:18:10.687031837Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-03T20:18:10.687288894Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-03T20:18:10.687324594Z INFO: Collecting host service logs for NetworkManager
2026-11-03T20:18:10.687657899Z INFO: Collecting host service logs for openvswitch
2026-11-03T20:18:10.687962967Z INFO: Collecting host service logs for ovs-configuration
2026-11-03T20:18:10.688263816Z INFO: Collecting host service logs for ovsdb-server
2026-11-03T20:18:10.688559073Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-03T20:18:10.688912400Z INFO: Waiting for worker host service log collection to complete ...
2026-11-03T20:18:10.696065882Z Executing Istio gather script
2026-11-03T20:18:10.926296284Z INFO: Waiting for node performance related collection to complete ...
2026-11-03T20:18:11.656789576Z No resources found in openshift-etcd namespace.
2026-11-03T20:18:11.759959982Z Wrote inspect data to must-gather.
2026-11-03T20:18:11.887027032Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-hjgvj
2026-11-03T20:18:11.887084578Z insights-runtime-extractor-bfp9j
2026-11-03T20:18:11.887107710Z insights-runtime-extractor-snkm9
2026-11-03T20:18:11.887128976Z insights-runtime-extractor-xsspg
2026-11-03T20:18:11.887253677Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-03T20:18:12.031348102Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-03T20:18:12.040758375Z No resources found
2026-11-03T20:18:12.101569118Z INFO: "metallb-operator" not detected. Skipping.
2026-11-03T20:18:12.134121637Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:12.191635318Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-03T20:18:12.191680453Z INFO: on-disk MachineConfig config collection complete.
2026-11-03T20:18:12.198854414Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-03T20:18:12.302070316Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-03T20:18:12.302070316Z See 'oc rsync -h' for help and examples
2026-11-03T20:18:12.413291128Z INFO: Worker host service log collection to complete.
2026-11-03T20:18:12.430879916Z error: the server doesn't have a resource type "performanceprofile"
2026-11-03T20:18:12.465271785Z INFO: "nfd" not detected. Skipping.
2026-11-03T20:18:12.500587161Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-03T20:18:12.514880921Z ERROR: No running kube-apiserver pods found
2026-11-03T20:18:12.568193874Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-03T20:18:12.813889921Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-03T20:18:12.881081644Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T20:18:13.023151386Z Wrote inspect data to must-gather.
2026-11-03T20:18:13.045103735Z Wrote inspect data to must-gather.
2026-11-03T20:18:13.238201036Z No resources found
2026-11-03T20:18:13.321407656Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-03T20:18:13.385918402Z Wrote inspect data to must-gather.
2026-11-03T20:18:13.439628333Z Wrote inspect data to must-gather.
2026-11-03T20:18:13.447176427Z INFO: 'previous-logs' folder not found on ip-10-0-128-89.us-west-2.compute.internal, skipping...
2026-11-03T20:18:13.538474643Z Wrote inspect data to must-gather.
2026-11-03T20:18:13.581173376Z Wrote inspect data to must-gather.
2026-11-03T20:18:13.717207387Z tar: Removing leading `/' from member names
2026-11-03T20:18:13.821547017Z INFO: HAProxy config collection complete.
2026-11-03T20:18:14.049527937Z error: the server doesn't have a resource type "machineconfigs"
2026-11-03T20:18:14.056780403Z INFO: 'previous-logs' folder not found on ip-10-0-138-213.us-west-2.compute.internal, skipping...
2026-11-03T20:18:14.199354853Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-03T20:18:14.337656943Z Wrote inspect data to must-gather.
2026-11-03T20:18:14.473633975Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-03T20:18:14.527854398Z INFO: 'previous-logs' folder not found on ip-10-0-142-155.us-west-2.compute.internal, skipping...
2026-11-03T20:18:14.527890588Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-03T20:18:14.527920517Z INFO: Machine Config Daemon termination log collection complete.
2026-11-03T20:18:14.900235097Z INFO: Getting rules from prometheus-k8s-0
2026-11-03T20:18:14.942948886Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-03T20:18:15.090648896Z Wrote inspect data to must-gather.
2026-11-03T20:18:15.157991835Z Wrote inspect data to must-gather.
2026-11-03T20:18:15.539953500Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-03T20:18:15.693335247Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T20:18:15.947720822Z Wrote inspect data to must-gather.
2026-11-03T20:18:15.982985166Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-03T20:18:16.199116913Z Wrote inspect data to must-gather.
2026-11-03T20:18:16.246073754Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T20:18:16.457632764Z Gathering data for ns/openshift-etcd...
2026-11-03T20:18:16.538074972Z INFO: Getting status/config from prometheus-k8s-0
2026-11-03T20:18:16.599915640Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:16.846731578Z INFO: LEGACY MODE
2026-11-03T20:18:16.846780251Z INFO: Gathering ovn-kubernetes master data
2026-11-03T20:18:16.894383990Z Wrote inspect data to must-gather.
2026-11-03T20:18:17.140648940Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:17.446341357Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:17.448961817Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-03T20:18:17.606821177Z Wrote inspect data to must-gather.
2026-11-03T20:18:17.719641502Z Gathering data for ns/openshift-monitoring...
2026-11-03T20:18:18.046486466Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:18.155738600Z Wrote inspect data to must-gather.
2026-11-03T20:18:18.344547545Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-03T20:18:18.358959561Z Wrote inspect data to must-gather.
2026-11-03T20:18:18.358996201Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T20:18:18.358996201Z     namespaces "assisted-installer" not found
2026-11-03T20:18:18.744544931Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:18.747775761Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-03T20:18:18.760777645Z Wrote inspect data to must-gather.
2026-11-03T20:18:19.314553388Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-03T20:18:19.375663263Z Wrote inspect data to must-gather.
2026-11-03T20:18:19.379644520Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:19.556434811Z INFO: Waiting for network log collection to complete ...
2026-11-03T20:18:19.556480026Z INFO: Waiting for ovnk database copies to complete ...
2026-11-03T20:18:19.556918059Z INFO: Copying ovnk databases complete.
2026-11-03T20:18:19.558480411Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-03T20:18:19.558992071Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 791 is not a child of this shell
2026-11-03T20:18:19.558992071Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 792 is not a child of this shell
2026-11-03T20:18:19.558992071Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 794 is not a child of this shell
2026-11-03T20:18:19.558992071Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 796 is not a child of this shell
2026-11-03T20:18:19.558992071Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 797 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 798 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 800 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 801 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 817 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 819 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 820 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 821 is not a child of this shell
2026-11-03T20:18:19.559033016Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 823 is not a child of this shell
2026-11-03T20:18:19.563226037Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-03T20:18:19.566743315Z tar: Exiting with failure status due to previous errors
2026-11-03T20:18:19.567276923Z INFO: Network log collection complete.
2026-11-03T20:18:19.984879676Z Wrote inspect data to must-gather.
2026-11-03T20:18:20.110240751Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:20.112760531Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-03T20:18:20.234443360Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-03T20:18:20.733523908Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:21.083509757Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-03T20:18:21.465553607Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:21.468213735Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-03T20:18:21.483823395Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-03T20:18:21.511673793Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-03T20:18:21.987615919Z daemonset.apps/perf-node-gather-daemonset created
2026-11-03T20:18:22.094581526Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:22.145758456Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:22.434388890Z Waiting for performance profile collector pods to become ready: 1
2026-11-03T20:18:22.820730188Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:22.823076942Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-11-03T20:18:22.823692363Z INFO: Getting status from alertmanager-main-0
2026-11-03T20:18:23.475552606Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:23.905045145Z Waiting for performance profile collector pods to become ready: 2
2026-11-03T20:18:24.203768546Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:24.206037072Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-11-03T20:18:24.811532183Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:25.339312140Z Waiting for performance profile collector pods to become ready: 3
2026-11-03T20:18:25.531692987Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:25.535040705Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-03T20:18:26.096288564Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:26.706912913Z Waiting for performance profile collector pods to become ready: 4
2026-11-03T20:18:26.817831607Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:18:27.150268897Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:28.043755234Z Waiting for performance profile collector pods to become ready: 5
2026-11-03T20:18:29.238034100Z Done executing Istio gather script
2026-11-03T20:18:29.403880625Z Waiting for performance profile collector pods to become ready: 6
2026-11-03T20:18:30.760558709Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-03T20:18:31.541191702Z Collecting performance related data for node ip-10-0-128-89.us-west-2.compute.internal
2026-11-03T20:18:31.548020177Z Collecting performance related data for node ip-10-0-142-155.us-west-2.compute.internal
2026-11-03T20:18:31.555081814Z Collecting performance related data for node ip-10-0-138-213.us-west-2.compute.internal
2026-11-03T20:18:32.156854770Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:37.163252221Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:37.723390822Z Gathering data for ns/openshift-network-console...
2026-11-03T20:18:40.246134824Z Gathering data for ns/openshift-console-operator...
2026-11-03T20:18:42.167631873Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:42.408821277Z Gathering data for ns/openshift-console...
2026-11-03T20:18:45.359903299Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-03T20:18:45.415840140Z Collecting kubelet logs for node ip-10-0-128-89.us-west-2.compute.internal
2026-11-03T20:18:45.417588911Z Collecting kubelet logs for node ip-10-0-142-155.us-west-2.compute.internal
2026-11-03T20:18:45.419242702Z Collecting kubelet logs for node ip-10-0-138-213.us-west-2.compute.internal
2026-11-03T20:18:46.432530341Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-cwq5w namespace
2026-11-03T20:18:46.484285781Z INFO: Node performance data collection complete.
2026-11-03T20:18:47.171820797Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:48.334947461Z Gathering data for ns/openshift-dns-operator...
2026-11-03T20:18:50.294694065Z Gathering data for ns/openshift-dns...
2026-11-03T20:18:52.176116293Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:55.959214604Z Gathering data for ns/openshift-image-registry...
2026-11-03T20:18:57.180359002Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:18:59.407362082Z Gathering data for ns/openshift-ingress-operator...
2026-11-03T20:19:01.261899459Z Gathering data for ns/openshift-ingress...
2026-11-03T20:19:02.184807250Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:03.528358050Z Gathering data for ns/openshift-ingress-canary...
2026-11-03T20:19:06.568136486Z Gathering data for ns/openshift-insights...
2026-11-03T20:19:07.189218314Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:12.193640915Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:15.300164185Z Gathering data for ns/kserve...
2026-11-03T20:19:17.198258008Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:20.127792789Z Gathering data for ns/openshift-config...
2026-11-03T20:19:21.831558919Z Gathering data for ns/openshift-config-managed...
2026-11-03T20:19:22.202641385Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:23.716897851Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-03T20:19:25.461657881Z Gathering data for ns/openshift-kube-apiserver...
2026-11-03T20:19:27.180667226Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-03T20:19:27.206937493Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:28.914687878Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-03T20:19:30.736389719Z Gathering data for ns/openshift-kube-scheduler...
2026-11-03T20:19:32.211842050Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:32.473839787Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-03T20:19:34.303830934Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-03T20:19:36.822764875Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-03T20:19:37.216298050Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:38.949163418Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-03T20:19:41.485866334Z Gathering data for ns/openshift-multus...
2026-11-03T20:19:42.220586580Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:47.224834335Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:52.229212019Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:19:57.048808530Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-03T20:19:57.233470970Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:02.237923751Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:07.242252407Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:09.796336826Z Gathering data for ns/openshift-host-network...
2026-11-03T20:20:11.788823924Z Gathering data for ns/openshift-network-diagnostics...
2026-11-03T20:20:12.246561764Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:16.105835458Z Gathering data for ns/openshift-network-node-identity...
2026-11-03T20:20:17.250926799Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:18.975529267Z Gathering data for ns/openshift-network-operator...
2026-11-03T20:20:21.950606725Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-03T20:20:22.255262118Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:23.668488111Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-03T20:20:27.021364101Z Gathering data for ns/openshift-apiserver-operator...
2026-11-03T20:20:27.259499139Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:28.762814181Z Gathering data for ns/openshift-apiserver...
2026-11-03T20:20:31.140775971Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-03T20:20:32.263732071Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:32.857233619Z Gathering data for ns/openshift-controller-manager...
2026-11-03T20:20:34.634067090Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-03T20:20:37.268209528Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:37.594105105Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-03T20:20:39.366927277Z Gathering data for ns/openshift-service-ca-operator...
2026-11-03T20:20:41.481085584Z Gathering data for ns/openshift-service-ca...
2026-11-03T20:20:42.272746705Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:43.937727428Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-03T20:20:47.277099344Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-11-03T20:20:49.650727761Z Wrote inspect data to must-gather.
2026-11-03T20:20:49.650766902Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T20:20:49.650766902Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-03T20:20:50.064534852Z error: the server doesn't have a resource type "clusters"
2026-11-03T20:20:50.108790347Z Caches written to disk