2026-11-03T21:27:18.186737204Z [disk usage checker] Started
2026-11-03T21:27:18.190739493Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:19.092175976Z Gathering data for ns/openshift-cluster-version...
2026-11-03T21:27:19.975558110Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T21:27:20.519267279Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T21:27:21.018921063Z Gathering data for ns/default...
2026-11-03T21:27:22.845724789Z Gathering data for ns/openshift...
2026-11-03T21:27:23.195462518Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:24.769080654Z Gathering data for ns/kube-system...
2026-11-03T21:27:25.022692316Z Waiting on subprocesses to finish execution.
2026-11-03T21:27:25.034162212Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-03T21:27:25.038634240Z Executing Istio gather script
2026-11-03T21:27:25.039199681Z INFO: Gathering HAProxy config files
2026-11-03T21:27:25.051750783Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-03T21:27:25.051844974Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-03T21:27:25.080115664Z INFO: Waiting for node performance related collection to complete ...
2026-11-03T21:27:25.170845574Z INFO: Collecting host service logs for crio
2026-11-03T21:27:25.171094115Z INFO: Collecting host service logs for kubelet
2026-11-03T21:27:25.171345241Z INFO: Collecting host service logs for rpm-ostreed
2026-11-03T21:27:25.171625761Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-03T21:27:25.171896729Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-03T21:27:25.172146563Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-03T21:27:25.172417281Z INFO: Collecting host service logs for NetworkManager
2026-11-03T21:27:25.172696191Z INFO: Collecting host service logs for openvswitch
2026-11-03T21:27:25.172948624Z INFO: Collecting host service logs for ovs-configuration
2026-11-03T21:27:25.173212658Z INFO: Collecting host service logs for ovsdb-server
2026-11-03T21:27:25.173457103Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-03T21:27:25.173787467Z INFO: Waiting for worker host service log collection to complete ...
2026-11-03T21:27:25.729315638Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-03T21:27:25.729362591Z INFO: on-disk MachineConfig config collection complete.
2026-11-03T21:27:25.751176641Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-krj8m
2026-11-03T21:27:25.751234948Z insights-runtime-extractor-4xn9r
2026-11-03T21:27:25.751254780Z insights-runtime-extractor-8hwjs
2026-11-03T21:27:25.751274705Z insights-runtime-extractor-gt9b8
2026-11-03T21:27:25.805678984Z No resources found in openshift-etcd namespace.
2026-11-03T21:27:25.824987505Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-03T21:27:26.165578425Z No resources found
2026-11-03T21:27:26.238431073Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-03T21:27:26.262335326Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-03T21:27:26.531351787Z error: the server doesn't have a resource type "performanceprofile"
2026-11-03T21:27:26.532553148Z INFO: "nfd" not detected. Skipping.
2026-11-03T21:27:26.551009924Z Wrote inspect data to must-gather.
2026-11-03T21:27:26.564683474Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-03T21:27:26.673313227Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-03T21:27:26.673313227Z See 'oc rsync -h' for help and examples
2026-11-03T21:27:26.760169857Z ERROR: No running kube-apiserver pods found
2026-11-03T21:27:26.760267668Z INFO: "metallb-operator" not detected. Skipping.
2026-11-03T21:27:26.772754437Z INFO: Worker host service log collection to complete.
2026-11-03T21:27:26.883069025Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-03T21:27:27.086589726Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-03T21:27:27.499561820Z Wrote inspect data to must-gather.
2026-11-03T21:27:27.542318674Z No resources found
2026-11-03T21:27:27.553864749Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-03T21:27:27.696069534Z tar: Removing leading `/' from member names
2026-11-03T21:27:27.699937627Z INFO: 'previous-logs' folder not found on ip-10-0-140-181.us-west-1.compute.internal, skipping...
2026-11-03T21:27:27.749996223Z INFO: HAProxy config collection complete.
2026-11-03T21:27:27.799990496Z Wrote inspect data to must-gather.
2026-11-03T21:27:27.838335155Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T21:27:27.921560962Z Wrote inspect data to must-gather.
2026-11-03T21:27:28.007870251Z error: the server doesn't have a resource type "machineconfigs"
2026-11-03T21:27:28.158768788Z INFO: 'previous-logs' folder not found on ip-10-0-140-215.us-west-1.compute.internal, skipping...
2026-11-03T21:27:28.172886130Z Wrote inspect data to must-gather.
2026-11-03T21:27:28.221968714Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:28.234075937Z Wrote inspect data to must-gather.
2026-11-03T21:27:28.334727926Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-03T21:27:28.467040220Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-03T21:27:28.571054790Z INFO: 'previous-logs' folder not found on ip-10-0-143-235.us-west-1.compute.internal, skipping...
2026-11-03T21:27:28.571099207Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-03T21:27:28.571111057Z INFO: Machine Config Daemon termination log collection complete.
2026-11-03T21:27:28.769829138Z Wrote inspect data to must-gather.
2026-11-03T21:27:28.955932310Z INFO: Getting rules from prometheus-k8s-0
2026-11-03T21:27:29.040655162Z Wrote inspect data to must-gather.
2026-11-03T21:27:29.044778053Z Wrote inspect data to must-gather.
2026-11-03T21:27:29.324496207Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-03T21:27:29.491007478Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-03T21:27:29.708586103Z Wrote inspect data to must-gather.
2026-11-03T21:27:29.832929676Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:29.909569457Z Wrote inspect data to must-gather.
2026-11-03T21:27:30.047401082Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-03T21:27:30.400137958Z INFO: Getting status/config from prometheus-k8s-0
2026-11-03T21:27:30.404236734Z Wrote inspect data to must-gather.
2026-11-03T21:27:30.457454830Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:30.461278735Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-03T21:27:30.517676570Z Wrote inspect data to must-gather.
2026-11-03T21:27:30.802188007Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T21:27:30.938380101Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:30.948060878Z Wrote inspect data to must-gather.
2026-11-03T21:27:31.099905364Z INFO: LEGACY MODE
2026-11-03T21:27:31.099945942Z INFO: Gathering ovn-kubernetes master data
2026-11-03T21:27:31.399408301Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T21:27:31.408331266Z Wrote inspect data to must-gather.
2026-11-03T21:27:31.612497147Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:31.614960391Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-03T21:27:31.828479778Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-03T21:27:31.840637379Z Wrote inspect data to must-gather.
2026-11-03T21:27:32.188920666Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:32.301588646Z Wrote inspect data to must-gather.
2026-11-03T21:27:32.711448924Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-03T21:27:32.835403095Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:32.838133947Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-03T21:27:32.885625635Z Gathering data for ns/openshift-etcd...
2026-11-03T21:27:32.926607518Z Wrote inspect data to must-gather.
2026-11-03T21:27:33.003233618Z Gathering data for ns/openshift-monitoring...
2026-11-03T21:27:33.193057407Z INFO: Waiting for network log collection to complete ...
2026-11-03T21:27:33.193099911Z INFO: Waiting for ovnk database copies to complete ...
2026-11-03T21:27:33.193616395Z INFO: Copying ovnk databases complete.
2026-11-03T21:27:33.195342186Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-03T21:27:33.195901219Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 802 is not a child of this shell
2026-11-03T21:27:33.195901219Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 805 is not a child of this shell
2026-11-03T21:27:33.195901219Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 807 is not a child of this shell
2026-11-03T21:27:33.195901219Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 808 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 809 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 811 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 812 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 813 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 865 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 866 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 867 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 868 is not a child of this shell
2026-11-03T21:27:33.195932518Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 869 is not a child of this shell
2026-11-03T21:27:33.197885798Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-03T21:27:33.201552393Z tar: Exiting with failure status due to previous errors
2026-11-03T21:27:33.202061936Z INFO: Network log collection complete.
2026-11-03T21:27:33.226628074Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:33.406535863Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:33.650850621Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-03T21:27:33.780936078Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-03T21:27:34.080705511Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:34.083756242Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-03T21:27:34.144548929Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-03T21:27:34.502739309Z daemonset.apps/perf-node-gather-daemonset created
2026-11-03T21:27:34.638639921Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:34.733202645Z Wrote inspect data to must-gather.
2026-11-03T21:27:34.733242666Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T21:27:34.733242666Z     namespaces "assisted-installer" not found
2026-11-03T21:27:34.785884064Z Waiting for performance profile collector pods to become ready: 1
2026-11-03T21:27:34.824193867Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-03T21:27:35.314797000Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:35.317500791Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-03T21:27:35.761866031Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:36.044171859Z Waiting for performance profile collector pods to become ready: 2
2026-11-03T21:27:36.054898337Z INFO: Getting status from alertmanager-main-0
2026-11-03T21:27:36.388219753Z Wrote inspect data to /must-gather/istio.
2026-11-03T21:27:37.296206890Z Waiting for performance profile collector pods to become ready: 3
2026-11-03T21:27:38.231410273Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:38.265656463Z Done executing Istio gather script
2026-11-03T21:27:38.549240849Z Waiting for performance profile collector pods to become ready: 4
2026-11-03T21:27:39.801214148Z Waiting for performance profile collector pods to become ready: 5
2026-11-03T21:27:41.055788847Z Waiting for performance profile collector pods to become ready: 6
2026-11-03T21:27:42.305809920Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-03T21:27:42.870334386Z Collecting performance related data for node ip-10-0-140-215.us-west-1.compute.internal
2026-11-03T21:27:42.877392345Z Collecting performance related data for node ip-10-0-143-235.us-west-1.compute.internal
2026-11-03T21:27:42.884799770Z Collecting performance related data for node ip-10-0-140-181.us-west-1.compute.internal
2026-11-03T21:27:43.236126070Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:48.240833563Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:53.245407196Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:54.157629232Z Gathering data for ns/openshift-network-console...
2026-11-03T21:27:56.824904831Z Gathering data for ns/openshift-console-operator...
2026-11-03T21:27:57.447543878Z Collecting kubelet logs for node ip-10-0-140-215.us-west-1.compute.internal
2026-11-03T21:27:57.449353326Z Collecting kubelet logs for node ip-10-0-143-235.us-west-1.compute.internal
2026-11-03T21:27:57.451730490Z Collecting kubelet logs for node ip-10-0-140-181.us-west-1.compute.internal
2026-11-03T21:27:58.250059672Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:27:58.326493593Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-5r6jx namespace
2026-11-03T21:27:58.382042881Z INFO: Node performance data collection complete.
2026-11-03T21:27:59.099865778Z Gathering data for ns/openshift-console...
2026-11-03T21:28:02.270646293Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-03T21:28:03.254926839Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:05.407247595Z Gathering data for ns/openshift-dns-operator...
2026-11-03T21:28:07.486484417Z Gathering data for ns/openshift-dns...
2026-11-03T21:28:08.259677322Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:13.264630311Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:13.548392927Z Gathering data for ns/openshift-image-registry...
2026-11-03T21:28:17.365615301Z Gathering data for ns/openshift-ingress-operator...
2026-11-03T21:28:18.269271369Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:19.362198722Z Gathering data for ns/openshift-ingress...
2026-11-03T21:28:21.773250191Z Gathering data for ns/openshift-ingress-canary...
2026-11-03T21:28:23.274099105Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:25.034368837Z Gathering data for ns/openshift-insights...
2026-11-03T21:28:28.278860514Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:33.283564950Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:33.389595296Z Gathering data for ns/openshift-kueue-operator...
2026-11-03T21:28:37.601228170Z Gathering data for ns/openshift-jobset-operator...
2026-11-03T21:28:38.288360129Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:40.933179717Z Gathering data for ns/openshift-config...
2026-11-03T21:28:42.790856103Z Gathering data for ns/openshift-config-managed...
2026-11-03T21:28:43.293034095Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:44.796624363Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-03T21:28:46.634628658Z Gathering data for ns/openshift-kube-apiserver...
2026-11-03T21:28:48.297974506Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:48.478809502Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-03T21:28:50.363877527Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-03T21:28:52.332433543Z Gathering data for ns/openshift-kube-scheduler...
2026-11-03T21:28:53.303020956Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:54.169849471Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-03T21:28:56.133236373Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-03T21:28:58.307875653Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:28:58.836855719Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-03T21:29:01.111378059Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-03T21:29:03.312560129Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:03.829211136Z Gathering data for ns/openshift-multus...
2026-11-03T21:29:08.317178013Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:13.321791606Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:18.326539645Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:20.525329649Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-03T21:29:23.331156706Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:28.335795211Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:33.340446964Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:34.361037954Z Gathering data for ns/openshift-host-network...
2026-11-03T21:29:36.497034464Z Gathering data for ns/openshift-network-diagnostics...
2026-11-03T21:29:38.349569629Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:41.176587498Z Gathering data for ns/openshift-network-node-identity...
2026-11-03T21:29:43.354540930Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:44.248547318Z Gathering data for ns/openshift-network-operator...
2026-11-03T21:29:47.465608708Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-03T21:29:48.359287753Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:49.302460765Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-03T21:29:52.929193332Z Gathering data for ns/openshift-apiserver-operator...
2026-11-03T21:29:53.363981285Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:54.791262270Z Gathering data for ns/openshift-apiserver...
2026-11-03T21:29:57.364666733Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-03T21:29:58.368707710Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:29:59.203669671Z Gathering data for ns/openshift-controller-manager...
2026-11-03T21:30:01.105208979Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-03T21:30:03.373398633Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:30:04.305191581Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-03T21:30:06.207874448Z Gathering data for ns/openshift-service-ca-operator...
2026-11-03T21:30:08.378521787Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:30:08.485742751Z Gathering data for ns/openshift-service-ca...
2026-11-03T21:30:11.112221841Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-03T21:30:13.383544811Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T21:30:17.300174908Z Wrote inspect data to must-gather.
2026-11-03T21:30:17.300221324Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T21:30:17.300221324Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-03T21:30:17.726383425Z error: the server doesn't have a resource type "clusters"
2026-11-03T21:30:17.772669830Z Caches written to disk