2026-11-02T16:17:28.518321711Z [disk usage checker] Started
2026-11-02T16:17:28.521512694Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:17:28.912734876Z Gathering data for ns/openshift-cluster-version...
2026-11-02T16:17:29.242154401Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-02T16:17:29.446919340Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-02T16:17:29.657266134Z Gathering data for ns/default...
2026-11-02T16:17:30.365216852Z Gathering data for ns/openshift...
2026-11-02T16:17:31.189980082Z Gathering data for ns/kube-system...
2026-11-02T16:17:33.525971729Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:17:33.706738790Z Gathering data for ns/openshift-etcd...
2026-11-02T16:17:34.417570384Z Wrote inspect data to must-gather.
2026-11-02T16:17:34.417601259Z error: inspection completed with the errors occurred while gathering data:
2026-11-02T16:17:34.417601259Z     namespaces "assisted-installer" not found
2026-11-02T16:17:38.530602584Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:17:41.872357437Z Waiting on subprocesses to finish execution.
2026-11-02T16:17:41.898733879Z INFO: Collecting host service logs for crio
2026-11-02T16:17:41.899041806Z INFO: Collecting host service logs for kubelet
2026-11-02T16:17:41.899345229Z INFO: Collecting host service logs for rpm-ostreed
2026-11-02T16:17:41.899800405Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-02T16:17:41.900223178Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-02T16:17:41.900637030Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-02T16:17:41.901688287Z Executing Istio gather script
2026-11-02T16:17:41.901742243Z INFO: Collecting host service logs for NetworkManager
2026-11-02T16:17:41.902066389Z INFO: Collecting host service logs for openvswitch
2026-11-02T16:17:41.902412941Z INFO: Collecting host service logs for ovs-configuration
2026-11-02T16:17:41.902851924Z INFO: Collecting host service logs for ovsdb-server
2026-11-02T16:17:41.903184508Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-02T16:17:41.903559546Z INFO: Waiting for worker host service log collection to complete ...
2026-11-02T16:17:41.914810549Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-02T16:17:41.916973076Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-02T16:17:41.939206937Z INFO: Gathering HAProxy config files
2026-11-02T16:17:41.945138594Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-02T16:17:41.957252184Z INFO: Waiting for node performance related collection to complete ...
2026-11-02T16:17:42.493189248Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-8nlz6
2026-11-02T16:17:42.493267338Z insights-runtime-extractor-jzm4s
2026-11-02T16:17:42.493293979Z insights-runtime-extractor-k6fjk
2026-11-02T16:17:42.493317795Z insights-runtime-extractor-rzg46
2026-11-02T16:17:42.921611799Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-02T16:17:43.142387987Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-02T16:17:43.239281646Z INFO: "metallb-operator" not detected. Skipping.
2026-11-02T16:17:43.317303852Z error: the server doesn't have a resource type "performanceprofile"
2026-11-02T16:17:43.384795909Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-02T16:17:43.384795909Z See 'oc rsync -h' for help and examples
2026-11-02T16:17:43.470950163Z No resources found
2026-11-02T16:17:43.505332639Z No resources found in openshift-etcd namespace.
2026-11-02T16:17:43.576920144Z INFO: "nfd" not detected. Skipping.
2026-11-02T16:17:43.611479405Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-02T16:17:43.699867790Z Wrote inspect data to must-gather.
2026-11-02T16:17:43.904185265Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:17:44.195527356Z Wrote inspect data to must-gather.
2026-11-02T16:17:44.356373252Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-02T16:17:44.468969918Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-02T16:17:44.475382691Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-02T16:17:44.564911594Z ERROR: No running kube-apiserver pods found
2026-11-02T16:17:44.568377809Z Wrote inspect data to must-gather.
2026-11-02T16:17:44.741687827Z No resources found
2026-11-02T16:17:44.811050831Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-02T16:17:44.904221157Z Wrote inspect data to must-gather.
2026-11-02T16:17:45.103022602Z Wrote inspect data to must-gather.
2026-11-02T16:17:45.125030985Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-02T16:17:45.184798543Z Wrote inspect data to must-gather.
2026-11-02T16:17:45.282921334Z INFO: HAProxy config collection complete.
2026-11-02T16:17:45.294493814Z Wrote inspect data to must-gather.
2026-11-02T16:17:45.301113244Z INFO: 'previous-logs' folder not found on ip-10-0-129-45.ec2.internal, skipping...
2026-11-02T16:17:45.325721433Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress
2026-11-02T16:17:45.677759720Z error: the server doesn't have a resource type "machineconfigs"
2026-11-02T16:17:45.709341806Z INFO: 'previous-logs' folder not found on ip-10-0-133-143.ec2.internal, skipping...
2026-11-02T16:17:45.775654664Z Wrote inspect data to must-gather.
2026-11-02T16:17:45.818576494Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:45.829047603Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress
2026-11-02T16:17:45.998033181Z INFO: Getting rules from prometheus-k8s-0
2026-11-02T16:17:46.090671324Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-02T16:17:46.106141496Z INFO: 'previous-logs' folder not found on ip-10-0-135-138.ec2.internal, skipping...
2026-11-02T16:17:46.106285376Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-02T16:17:46.106358119Z INFO: Machine Config Daemon termination log collection complete.
2026-11-02T16:17:46.310641367Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:46.314386973Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress
2026-11-02T16:17:46.338753081Z Wrote inspect data to must-gather.
2026-11-02T16:17:46.500980989Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-02T16:17:46.511833399Z Wrote inspect data to must-gather.
2026-11-02T16:17:46.726940414Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:46.730666351Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress
2026-11-02T16:17:46.793473244Z Wrote inspect data to must-gather.
2026-11-02T16:17:46.809978265Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-02T16:17:47.185029473Z Wrote inspect data to must-gather.
2026-11-02T16:17:47.219357222Z INFO: LEGACY MODE
2026-11-02T16:17:47.219415631Z INFO: Gathering ovn-kubernetes master data
2026-11-02T16:17:47.469678710Z Wrote inspect data to must-gather.
2026-11-02T16:17:47.808484596Z Wrote inspect data to must-gather.
2026-11-02T16:17:48.145862704Z Wrote inspect data to must-gather.
2026-11-02T16:17:48.300075824Z INFO: Waiting for network log collection to complete ...
2026-11-02T16:17:48.300119982Z INFO: Waiting for ovnk database copies to complete ...
2026-11-02T16:17:48.450868546Z Wrote inspect data to must-gather.
2026-11-02T16:17:48.719671556Z Wrote inspect data to must-gather.
2026-11-02T16:17:48.912427189Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:17:49.005522842Z Wrote inspect data to must-gather.
2026-11-02T16:17:49.370723168Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-02T16:17:49.538747230Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-02T16:17:49.703700185Z daemonset.apps/perf-node-gather-daemonset created
2026-11-02T16:17:49.860074906Z Waiting for performance profile collector pods to become ready: 1
2026-11-02T16:17:50.427336276Z Wrote inspect data to must-gather.
2026-11-02T16:17:50.995377182Z Waiting for performance profile collector pods to become ready: 2
2026-11-02T16:17:52.136069293Z Waiting for performance profile collector pods to become ready: 3
2026-11-02T16:17:53.284909470Z Waiting for performance profile collector pods to become ready: 4
2026-11-02T16:17:53.332710007Z Unable to connect to the server: net/http: TLS handshake timeout
2026-11-02T16:17:53.334757153Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-02T16:17:53.334783964Z INFO: on-disk MachineConfig config collection complete.
2026-11-02T16:17:53.344845511Z error: Get "https://172.31.0.1:443/api/v1/nodes?labelSelector=node-role.kubernetes.io%2Fmaster": net/http: TLS handshake timeout
2026-11-02T16:17:53.347086647Z INFO: Worker host service log collection to complete.
2026-11-02T16:17:53.916854792Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:17:54.288341735Z Unable to connect to the server: net/http: TLS handshake timeout
2026-11-02T16:17:54.293475178Z INFO: Copying ovnk databases complete.
2026-11-02T16:17:54.294893687Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-02T16:17:54.295387688Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 739 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 740 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 741 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 742 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 744 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 745 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 746 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 747 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 789 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 790 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 792 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 794 is not a child of this shell
2026-11-02T16:17:54.295436958Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 795 is not a child of this shell
2026-11-02T16:17:54.296641491Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-02T16:17:54.300538692Z tar: Exiting with failure status due to previous errors
2026-11-02T16:17:54.301057661Z INFO: Network log collection complete.
2026-11-02T16:17:54.423995541Z Waiting for performance profile collector pods to become ready: 5
2026-11-02T16:17:56.698340642Z Unable to connect to the server: net/http: TLS handshake timeout
2026-11-02T16:17:56.700891299Z /usr/bin/monitoring_common.sh: line 15: READY_PROM_PODS[0]: unbound variable
2026-11-02T16:17:56.704405393Z INFO: Getting status/config from 
2026-11-02T16:17:56.867837274Z E1102 16:17:56.867795    1231 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://172.31.0.1:443/api?timeout=32s\": net/http: TLS handshake timeout"
2026-11-02T16:17:56.952972411Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-02T16:17:57.043634817Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:57.046473911Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress
2026-11-02T16:17:57.296694278Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-02T16:17:57.313196822Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:57.316161837Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress
2026-11-02T16:17:57.578127959Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:57.649860100Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-02T16:17:58.069176762Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-02T16:17:58.582416009Z INFO: Getting status from alertmanager-main-0
2026-11-02T16:17:58.921630704Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:17:59.447524624Z Inspecting resource crd/authorizationpolicies.security.istio.io
2026-11-02T16:17:59.673623217Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:59.951786307Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:17:59.955061235Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-02T16:18:00.203844159Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:00.465738157Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:00.469106112Z Inspecting resource crd/destinationrules.networking.istio.io
2026-11-02T16:18:00.741818388Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:01.007740985Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:01.010337120Z Inspecting resource crd/envoyfilters.networking.istio.io
2026-11-02T16:18:01.236813609Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:01.566031857Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:01.569042268Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-02T16:18:01.792741711Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:02.049971437Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:02.052829382Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-02T16:18:02.292516647Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:02.559241085Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:02.561928502Z Inspecting resource crd/gateways.networking.istio.io
2026-11-02T16:18:02.801117996Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:03.074302436Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:03.077582956Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-02T16:18:03.306255934Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:03.580341578Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:03.583136450Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-02T16:18:03.855108661Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:03.926454829Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:18:04.161148407Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:04.163797912Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-11-02T16:18:04.381177184Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:04.635183668Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:04.638081652Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-11-02T16:18:04.855759484Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:05.124655699Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:05.127862904Z Inspecting resource crd/peerauthentications.security.istio.io
2026-11-02T16:18:05.351971878Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:05.512700075Z Unable to connect to the server: net/http: TLS handshake timeout
2026-11-02T16:18:05.515932588Z Daemonset perf-node-gather-daemonset ready  out of 
2026-11-02T16:18:05.640792477Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:05.643437063Z Inspecting resource crd/proxyconfigs.networking.istio.io
2026-11-02T16:18:05.851705971Z Collecting performance related data for node ip-10-0-129-45.ec2.internal
2026-11-02T16:18:05.861050535Z Collecting performance related data for node ip-10-0-133-143.ec2.internal
2026-11-02T16:18:05.864798405Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:05.868317096Z Collecting performance related data for node ip-10-0-135-138.ec2.internal
2026-11-02T16:18:06.186986515Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:06.189635734Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-02T16:18:06.410852644Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:06.709320643Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:06.712853163Z Inspecting resource crd/requestauthentications.security.istio.io
2026-11-02T16:18:06.972313093Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:07.262704263Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:07.265194713Z Inspecting resource crd/serviceentries.networking.istio.io
2026-11-02T16:18:07.518589789Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:07.799162754Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:07.801852294Z Inspecting resource crd/sidecars.networking.istio.io
2026-11-02T16:18:08.058212231Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:08.344201567Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:08.347127544Z Inspecting resource crd/telemetries.telemetry.istio.io
2026-11-02T16:18:08.582949859Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:08.843013577Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:08.845596039Z Inspecting resource crd/virtualservices.networking.istio.io
2026-11-02T16:18:08.931233645Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:18:09.121065204Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:09.417214784Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:09.419835577Z Inspecting resource crd/wasmplugins.extensions.istio.io
2026-11-02T16:18:09.637604187Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:09.901218485Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:09.903982786Z Inspecting resource crd/workloadentries.networking.istio.io
2026-11-02T16:18:10.153505154Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:10.431627351Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:10.434855605Z Inspecting resource crd/workloadgroups.networking.istio.io
2026-11-02T16:18:10.664664652Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:10.931200105Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:11.070556628Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress
2026-11-02T16:18:11.351167475Z Gathering data for ns/openshift-ingress...
2026-11-02T16:18:13.935797997Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:18:17.414961683Z Unable to connect to the server: net/http: TLS handshake timeout
2026-11-02T16:18:17.904132701Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-02T16:18:18.940251416Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:18:19.999189464Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-02T16:18:20.981324338Z Wrote inspect data to /must-gather/istio.
2026-11-02T16:18:23.800838285Z Collecting kubelet logs for node ip-10-0-129-45.ec2.internal
2026-11-02T16:18:23.802609851Z Collecting kubelet logs for node ip-10-0-133-143.ec2.internal
2026-11-02T16:18:23.804434394Z Collecting kubelet logs for node ip-10-0-135-138.ec2.internal
2026-11-02T16:18:23.944841746Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:18:24.666342063Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-xl725 namespace
2026-11-02T16:18:24.685104845Z INFO: Node performance data collection complete.
2026-11-02T16:18:28.949569506Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-11-02T16:18:31.078430966Z Unable to connect to the server: net/http: TLS handshake timeout
2026-11-02T16:18:31.848183866Z Done executing Istio gather script
2026-11-02T16:18:32.030392376Z error: the server doesn't have a resource type "clusters"
2026-11-02T16:18:32.132608798Z Caches written to disk