2026-11-04T04:29:25.862219699Z [disk usage checker] Started
2026-11-04T04:29:25.865424132Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:29:26.870333618Z Gathering data for ns/openshift-cluster-version...
2026-11-04T04:29:27.759210250Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T04:29:28.332527702Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T04:29:28.872430337Z Gathering data for ns/default...
2026-11-04T04:29:30.720290689Z Gathering data for ns/openshift...
2026-11-04T04:29:30.873144807Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:29:32.683544913Z Gathering data for ns/kube-system...
2026-11-04T04:29:32.813974165Z Waiting on subprocesses to finish execution.
2026-11-04T04:29:32.822097501Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-04T04:29:32.823148828Z Executing Istio gather script
2026-11-04T04:29:32.827887967Z INFO: Gathering HAProxy config files
2026-11-04T04:29:32.831105345Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-04T04:29:32.831479596Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-04T04:29:32.872551629Z INFO: Waiting for node performance related collection to complete ...
2026-11-04T04:29:32.890321915Z INFO: Collecting host service logs for crio
2026-11-04T04:29:32.890662129Z INFO: Collecting host service logs for kubelet
2026-11-04T04:29:32.890979437Z INFO: Collecting host service logs for rpm-ostreed
2026-11-04T04:29:32.891303574Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-04T04:29:32.891640969Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-04T04:29:32.891950512Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-04T04:29:32.892257526Z INFO: Collecting host service logs for NetworkManager
2026-11-04T04:29:32.892574272Z INFO: Collecting host service logs for openvswitch
2026-11-04T04:29:32.892896980Z INFO: Collecting host service logs for ovs-configuration
2026-11-04T04:29:32.893189422Z INFO: Collecting host service logs for ovsdb-server
2026-11-04T04:29:32.893475969Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-04T04:29:32.894295116Z INFO: Waiting for worker host service log collection to complete ...
2026-11-04T04:29:33.827750921Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-jpsjl
2026-11-04T04:29:33.827817402Z insights-runtime-extractor-bfg4d
2026-11-04T04:29:33.827837479Z insights-runtime-extractor-jztz8
2026-11-04T04:29:33.827858802Z insights-runtime-extractor-t9g89
2026-11-04T04:29:33.961447253Z error: the server doesn't have a resource type "performanceprofile"
2026-11-04T04:29:34.251038979Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-04T04:29:34.259270838Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-04T04:29:34.282881805Z No resources found
2026-11-04T04:29:34.299850095Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-04T04:29:34.731150860Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-04T04:29:34.731188059Z INFO: on-disk MachineConfig config collection complete.
2026-11-04T04:29:34.797160940Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-04T04:29:34.797160940Z See 'oc rsync -h' for help and examples
2026-11-04T04:29:34.839629469Z Wrote inspect data to must-gather.
2026-11-04T04:29:34.848861682Z INFO: "nfd" not detected. Skipping.
2026-11-04T04:29:34.982384965Z No resources found in openshift-etcd namespace.
2026-11-04T04:29:35.009417879Z INFO: Worker host service log collection to complete.
2026-11-04T04:29:35.048809627Z INFO: "metallb-operator" not detected. Skipping.
2026-11-04T04:29:35.296970672Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T04:29:35.355639988Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-04T04:29:35.359599610Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-04T04:29:35.390278185Z Wrote inspect data to must-gather.
2026-11-04T04:29:35.395927647Z No resources found
2026-11-04T04:29:35.650021186Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-04T04:29:35.703020599Z ERROR: No running kube-apiserver pods found
2026-11-04T04:29:35.773534488Z INFO: 'previous-logs' folder not found on ip-10-0-128-165.us-west-2.compute.internal, skipping...
2026-11-04T04:29:35.798735504Z Wrote inspect data to must-gather.
2026-11-04T04:29:35.823351391Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-04T04:29:35.879232753Z Wrote inspect data to must-gather.
2026-11-04T04:29:35.889631432Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:29:35.904399101Z Wrote inspect data to must-gather.
2026-11-04T04:29:36.138249610Z Wrote inspect data to must-gather.
2026-11-04T04:29:36.262261336Z INFO: 'previous-logs' folder not found on ip-10-0-128-244.us-west-2.compute.internal, skipping...
2026-11-04T04:29:36.343625927Z error: the server doesn't have a resource type "machineconfigs"
2026-11-04T04:29:36.517682389Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-04T04:29:36.652841533Z INFO: 'previous-logs' folder not found on ip-10-0-135-81.us-west-2.compute.internal, skipping...
2026-11-04T04:29:36.652886966Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-04T04:29:36.652902605Z INFO: Machine Config Daemon termination log collection complete.
2026-11-04T04:29:36.728677586Z Wrote inspect data to must-gather.
2026-11-04T04:29:36.785367354Z Wrote inspect data to must-gather.
2026-11-04T04:29:36.787675594Z tar: Removing leading `/' from member names
2026-11-04T04:29:36.877010126Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-04T04:29:36.895625549Z INFO: HAProxy config collection complete.
2026-11-04T04:29:36.966804549Z INFO: Getting rules from prometheus-k8s-0
2026-11-04T04:29:37.411626985Z Wrote inspect data to must-gather.
2026-11-04T04:29:37.431527943Z Wrote inspect data to must-gather.
2026-11-04T04:29:37.673800113Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-04T04:29:37.891222878Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-04T04:29:37.961472529Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-04T04:29:38.041300112Z Wrote inspect data to must-gather.
2026-11-04T04:29:38.235532900Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:38.338406261Z Wrote inspect data to must-gather.
2026-11-04T04:29:38.509915850Z INFO: Getting status/config from prometheus-k8s-0
2026-11-04T04:29:38.706043326Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-04T04:29:38.743331919Z INFO: LEGACY MODE
2026-11-04T04:29:38.743372239Z INFO: Gathering ovn-kubernetes master data
2026-11-04T04:29:38.763292157Z Wrote inspect data to must-gather.
2026-11-04T04:29:38.843950793Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:38.848198337Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-04T04:29:39.204705399Z Wrote inspect data to must-gather.
2026-11-04T04:29:39.261537384Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-04T04:29:39.298886441Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:39.621448424Z Wrote inspect data to must-gather.
2026-11-04T04:29:39.980353498Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-04T04:29:40.008177902Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:40.012652387Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-04T04:29:40.245336730Z Wrote inspect data to must-gather.
2026-11-04T04:29:40.579746653Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:40.725701804Z Wrote inspect data to must-gather.
2026-11-04T04:29:40.774485514Z Gathering data for ns/openshift-monitoring...
2026-11-04T04:29:40.782370842Z INFO: Waiting for network log collection to complete ...
2026-11-04T04:29:40.782400666Z INFO: Waiting for ovnk database copies to complete ...
2026-11-04T04:29:40.782897273Z INFO: Copying ovnk databases complete.
2026-11-04T04:29:40.784807732Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-04T04:29:40.785249539Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 696 is not a child of this shell
2026-11-04T04:29:40.785249539Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 697 is not a child of this shell
2026-11-04T04:29:40.785249539Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 698 is not a child of this shell
2026-11-04T04:29:40.785249539Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 699 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 700 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 701 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 702 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 703 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 719 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 720 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 721 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 722 is not a child of this shell
2026-11-04T04:29:40.785270343Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 723 is not a child of this shell
2026-11-04T04:29:40.786693948Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-04T04:29:40.790003479Z tar: Exiting with failure status due to previous errors
2026-11-04T04:29:40.790536032Z INFO: Network log collection complete.
2026-11-04T04:29:40.897917964Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:29:40.928673389Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-04T04:29:40.966295468Z Gathering data for ns/openshift-etcd...
2026-11-04T04:29:41.229283464Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:41.232776248Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-04T04:29:41.338082209Z Wrote inspect data to must-gather.
2026-11-04T04:29:41.762888888Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-04T04:29:41.765663381Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:42.139718779Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-04T04:29:42.418790419Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:42.422103003Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-04T04:29:42.521609705Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-04T04:29:42.820540281Z Wrote inspect data to must-gather.
2026-11-04T04:29:42.820577639Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T04:29:42.820577639Z     namespaces "assisted-installer" not found
2026-11-04T04:29:42.881204968Z daemonset.apps/perf-node-gather-daemonset created
2026-11-04T04:29:42.952873966Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:42.967940728Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-04T04:29:43.137599954Z Waiting for performance profile collector pods to become ready: 1
2026-11-04T04:29:43.614368154Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:43.617886137Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-04T04:29:44.052422050Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:44.219694101Z INFO: Getting status from alertmanager-main-0
2026-11-04T04:29:44.397936882Z Waiting for performance profile collector pods to become ready: 2
2026-11-04T04:29:44.644330331Z Wrote inspect data to /must-gather/istio.
2026-11-04T04:29:45.660474135Z Waiting for performance profile collector pods to become ready: 3
2026-11-04T04:29:45.902675539Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:29:46.583822486Z Done executing Istio gather script
2026-11-04T04:29:46.913724409Z Waiting for performance profile collector pods to become ready: 4
2026-11-04T04:29:48.172847741Z Waiting for performance profile collector pods to become ready: 5
2026-11-04T04:29:49.431700140Z Waiting for performance profile collector pods to become ready: 6
2026-11-04T04:29:50.676452761Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-04T04:29:50.907077089Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:29:51.232117949Z Collecting performance related data for node ip-10-0-135-81.us-west-2.compute.internal
2026-11-04T04:29:51.239011802Z Collecting performance related data for node ip-10-0-128-165.us-west-2.compute.internal
2026-11-04T04:29:51.246041297Z Collecting performance related data for node ip-10-0-128-244.us-west-2.compute.internal
2026-11-04T04:29:55.911368209Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:00.915746326Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:01.561381975Z Gathering data for ns/openshift-network-console...
2026-11-04T04:30:04.067996858Z Gathering data for ns/openshift-console-operator...
2026-11-04T04:30:05.303163424Z Collecting kubelet logs for node ip-10-0-135-81.us-west-2.compute.internal
2026-11-04T04:30:05.304907119Z Collecting kubelet logs for node ip-10-0-128-165.us-west-2.compute.internal
2026-11-04T04:30:05.306655194Z Collecting kubelet logs for node ip-10-0-128-244.us-west-2.compute.internal
2026-11-04T04:30:05.921299705Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:06.224407631Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-9h74w namespace
2026-11-04T04:30:06.260098775Z Gathering data for ns/openshift-console...
2026-11-04T04:30:06.277069788Z INFO: Node performance data collection complete.
2026-11-04T04:30:09.234271984Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-04T04:30:10.925966086Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:12.180874953Z Gathering data for ns/openshift-dns-operator...
2026-11-04T04:30:14.137298922Z Gathering data for ns/openshift-dns...
2026-11-04T04:30:15.930274128Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:19.969262101Z Gathering data for ns/openshift-image-registry...
2026-11-04T04:30:20.934549217Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:23.518562651Z Gathering data for ns/openshift-ingress-operator...
2026-11-04T04:30:25.381484339Z Gathering data for ns/openshift-ingress...
2026-11-04T04:30:25.940325825Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:27.649109317Z Gathering data for ns/openshift-ingress-canary...
2026-11-04T04:30:30.750730533Z Gathering data for ns/openshift-insights...
2026-11-04T04:30:30.944585268Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:35.948969588Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:39.134166302Z Gathering data for ns/openshift-kueue-operator...
2026-11-04T04:30:40.953410723Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:43.255480472Z Gathering data for ns/openshift-jobset-operator...
2026-11-04T04:30:45.957849078Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:46.446618848Z Gathering data for ns/openshift-config...
2026-11-04T04:30:48.170585465Z Gathering data for ns/openshift-config-managed...
2026-11-04T04:30:50.051962685Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-04T04:30:50.962136062Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:51.768928335Z Gathering data for ns/openshift-kube-apiserver...
2026-11-04T04:30:53.487890616Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-04T04:30:55.234893571Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-04T04:30:55.966629240Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:30:57.081135346Z Gathering data for ns/openshift-kube-scheduler...
2026-11-04T04:30:58.805534998Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-04T04:31:00.655203043Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-04T04:31:00.971055698Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:03.195234662Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-04T04:31:05.334210806Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-04T04:31:05.975475024Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:07.899932988Z Gathering data for ns/openshift-multus...
2026-11-04T04:31:10.980016620Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:15.984465189Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:20.988872166Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:24.276584373Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-04T04:31:25.993434394Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:30.997868666Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:36.002275552Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:37.475937465Z Gathering data for ns/openshift-host-network...
2026-11-04T04:31:39.480301252Z Gathering data for ns/openshift-network-diagnostics...
2026-11-04T04:31:41.007299428Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:43.882239199Z Gathering data for ns/openshift-network-node-identity...
2026-11-04T04:31:46.013099068Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:46.770356432Z Gathering data for ns/openshift-network-operator...
2026-11-04T04:31:49.824022484Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-04T04:31:51.017571480Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:51.550738724Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-04T04:31:54.976843026Z Gathering data for ns/openshift-apiserver-operator...
2026-11-04T04:31:56.021910614Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:31:56.720243378Z Gathering data for ns/openshift-apiserver...
2026-11-04T04:31:59.122577027Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-04T04:32:00.838869545Z Gathering data for ns/openshift-controller-manager...
2026-11-04T04:32:01.026245546Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:32:02.614035941Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-04T04:32:05.631582304Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-04T04:32:06.030553858Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:32:07.420110550Z Gathering data for ns/openshift-service-ca-operator...
2026-11-04T04:32:09.564935527Z Gathering data for ns/openshift-service-ca...
2026-11-04T04:32:11.035599310Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:32:12.048258573Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-04T04:32:16.040130764Z [disk usage checker] Volume usage percentage: current = 63 ; allowed = 70
2026-11-04T04:32:18.026194453Z Wrote inspect data to must-gather.
2026-11-04T04:32:18.026241315Z error: inspection completed with the errors occurred while gathering data:
2026-11-04T04:32:18.026241315Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-04T04:32:18.445230863Z error: the server doesn't have a resource type "clusters"
2026-11-04T04:32:18.489616734Z Caches written to disk