2026-11-03T20:17:48.310781919Z [disk usage checker] Started
2026-11-03T20:17:48.314147375Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T20:17:48.520506107Z Gathering data for ns/openshift-cluster-version...
2026-11-03T20:17:48.662131258Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T20:17:48.767766740Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T20:17:48.884596660Z Gathering data for ns/default...
2026-11-03T20:17:49.278858423Z Gathering data for ns/openshift...
2026-11-03T20:17:49.771682152Z Gathering data for ns/kube-system...
2026-11-03T20:17:50.720736694Z Gathering data for ns/openshift-etcd...
2026-11-03T20:17:51.143118422Z Wrote inspect data to must-gather.
2026-11-03T20:17:51.143154430Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T20:17:51.143154430Z     namespaces "assisted-installer" not found
2026-11-03T20:17:51.192667311Z Waiting on subprocesses to finish execution.
2026-11-03T20:17:51.198599110Z INFO: Gathering machine config daemon's old logs from all nodes
2026-11-03T20:17:51.203744895Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-11-03T20:17:51.205676890Z Executing Istio gather script
2026-11-03T20:17:51.227112390Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-11-03T20:17:51.231616903Z INFO: Gathering HAProxy config files
2026-11-03T20:17:51.247191602Z INFO: Collecting host service logs for crio
2026-11-03T20:17:51.247585141Z INFO: Collecting host service logs for kubelet
2026-11-03T20:17:51.247932302Z INFO: Collecting host service logs for rpm-ostreed
2026-11-03T20:17:51.248260685Z INFO: Collecting host service logs for ostree-finalize-staged
2026-11-03T20:17:51.248616238Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-11-03T20:17:51.248944940Z INFO: Collecting host service logs for machine-config-daemon-host
2026-11-03T20:17:51.249266477Z INFO: Collecting host service logs for NetworkManager
2026-11-03T20:17:51.249614056Z INFO: Collecting host service logs for openvswitch
2026-11-03T20:17:51.249968017Z INFO: Collecting host service logs for ovs-configuration
2026-11-03T20:17:51.250200826Z INFO: Waiting for node performance related collection to complete ...
2026-11-03T20:17:51.250328339Z INFO: Collecting host service logs for ovsdb-server
2026-11-03T20:17:51.250662660Z INFO: Collecting host service logs for ovs-vswitchd
2026-11-03T20:17:51.251023464Z INFO: Waiting for worker host service log collection to complete ...
2026-11-03T20:17:51.834626914Z INFO: Collecting Insights Archives from insights-operator-7c4986d449-m5zxg
2026-11-03T20:17:51.834708585Z insights-runtime-extractor-8vgm9
2026-11-03T20:17:51.834731878Z insights-runtime-extractor-gtn5g
2026-11-03T20:17:51.834751897Z insights-runtime-extractor-nbjh4
2026-11-03T20:17:52.112673293Z INFO: "sriov-network-operator" not detected. Skipping.
2026-11-03T20:17:52.121105576Z INFO: "metallb-operator" not detected. Skipping.
2026-11-03T20:17:52.121528324Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-11-03T20:17:52.121603153Z INFO: on-disk MachineConfig config collection complete.
2026-11-03T20:17:52.377329624Z Wrote inspect data to must-gather.
2026-11-03T20:17:52.409021192Z No resources found
2026-11-03T20:17:52.522307416Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-11-03T20:17:52.522869285Z error: the server doesn't have a resource type "performanceprofile"
2026-11-03T20:17:52.540515839Z INFO: Found 1 replicas - prometheus-k8s-0
2026-11-03T20:17:52.557035184Z No resources found in openshift-etcd namespace.
2026-11-03T20:17:52.577601991Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-11-03T20:17:52.577601991Z See 'oc rsync -h' for help and examples
2026-11-03T20:17:52.618033497Z INFO: "nfd" not detected. Skipping.
2026-11-03T20:17:52.857049840Z INFO: Waiting for HAProxy config collection to complete ...
2026-11-03T20:17:52.891304422Z ERROR: No running kube-apiserver pods found
2026-11-03T20:17:52.937156321Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-11-03T20:17:52.958307540Z INFO: Worker host service log collection to complete.
2026-11-03T20:17:53.036601547Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-11-03T20:17:53.192536576Z Wrote inspect data to must-gather.
2026-11-03T20:17:53.440424534Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T20:17:53.816390700Z No resources found
2026-11-03T20:17:53.852623432Z Wrote inspect data to must-gather.
2026-11-03T20:17:53.861277456Z tar: Removing leading `/' from member names
2026-11-03T20:17:53.878239203Z INFO: HAProxy config collection complete.
2026-11-03T20:17:53.974979274Z Wrote inspect data to must-gather.
2026-11-03T20:17:54.054176357Z Wrote inspect data to must-gather.
2026-11-03T20:17:54.088144874Z Wrote inspect data to must-gather.
2026-11-03T20:17:54.101541668Z error: the server doesn't have a resource type "machineconfigs"
2026-11-03T20:17:54.109268871Z INFO: 'previous-logs' folder not found on ip-10-0-128-73.us-east-2.compute.internal, skipping...
2026-11-03T20:17:54.199752689Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T20:17:54.308071248Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-11-03T20:17:54.372938955Z Wrote inspect data to must-gather.
2026-11-03T20:17:54.446808959Z error: the server doesn't have a resource type "machineconfigpools"
2026-11-03T20:17:54.507355489Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-11-03T20:17:54.515827103Z INFO: Getting rules from prometheus-k8s-0
2026-11-03T20:17:54.550742683Z INFO: 'previous-logs' folder not found on ip-10-0-140-232.us-east-2.compute.internal, skipping...
2026-11-03T20:17:54.578908664Z Wrote inspect data to must-gather.
2026-11-03T20:17:54.582196715Z Gathering data for ns/openshift-cloud-credential-operator...
2026-11-03T20:17:54.762001031Z Wrote inspect data to must-gather.
2026-11-03T20:17:54.797256717Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable no sooner than v6.0+
2026-11-03T20:17:54.865765383Z INFO: 'previous-logs' folder not found on ip-10-0-140-51.us-east-2.compute.internal, skipping...
2026-11-03T20:17:54.865806425Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-11-03T20:17:54.865838168Z INFO: Machine Config Daemon termination log collection complete.
2026-11-03T20:17:54.918026787Z Wrote inspect data to must-gather.
2026-11-03T20:17:54.968500895Z error: the server doesn't have a resource type "kubeletconfigs"
2026-11-03T20:17:54.977641065Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-11-03T20:17:55.079748066Z INFO: Getting status/config from prometheus-k8s-0
2026-11-03T20:17:55.285502727Z Wrote inspect data to must-gather.
2026-11-03T20:17:55.289876714Z Wrote inspect data to must-gather.
2026-11-03T20:17:55.478142958Z Gathering data for ns/openshift-monitoring...
2026-11-03T20:17:55.569726850Z Wrote inspect data to must-gather.
2026-11-03T20:17:55.602902644Z INFO: Getting status/flags from prometheus-k8s-0
2026-11-03T20:17:55.659912926Z INFO: LEGACY MODE
2026-11-03T20:17:55.659945915Z INFO: Gathering ovn-kubernetes master data
2026-11-03T20:17:55.819499891Z Wrote inspect data to must-gather.
2026-11-03T20:17:55.832847900Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-11-03T20:17:56.077822383Z Wrote inspect data to must-gather.
2026-11-03T20:17:56.098793985Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-11-03T20:17:56.368738814Z Wrote inspect data to must-gather.
2026-11-03T20:17:56.373430736Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-11-03T20:17:56.446223786Z Inspecting resource crd/backendtlspolicies.gateway.networking.k8s.io
2026-11-03T20:17:56.585109613Z INFO: Waiting for network log collection to complete ...
2026-11-03T20:17:56.585146080Z INFO: Waiting for ovnk database copies to complete ...
2026-11-03T20:17:56.585705483Z INFO: Copying ovnk databases complete.
2026-11-03T20:17:56.587529267Z du: cannot access 'must-gather/network_logs/ovnk_database_store': No such file or directory
2026-11-03T20:17:56.588043824Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 804 is not a child of this shell
2026-11-03T20:17:56.588043824Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 805 is not a child of this shell
2026-11-03T20:17:56.588043824Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 806 is not a child of this shell
2026-11-03T20:17:56.588064646Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 807 is not a child of this shell
2026-11-03T20:17:56.588064646Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 809 is not a child of this shell
2026-11-03T20:17:56.588064646Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 810 is not a child of this shell
2026-11-03T20:17:56.588064646Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 811 is not a child of this shell
2026-11-03T20:17:56.588064646Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 812 is not a child of this shell
2026-11-03T20:17:56.588064646Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 831 is not a child of this shell
2026-11-03T20:17:56.588077139Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 832 is not a child of this shell
2026-11-03T20:17:56.588077139Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 834 is not a child of this shell
2026-11-03T20:17:56.588077139Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 835 is not a child of this shell
2026-11-03T20:17:56.588077139Z /usr/bin/gather_network_logs_basics: line 190: wait: pid 837 is not a child of this shell
2026-11-03T20:17:56.591467516Z tar: ovnk_database_store: Cannot stat: No such file or directory
2026-11-03T20:17:56.595267356Z tar: Exiting with failure status due to previous errors
2026-11-03T20:17:56.595898139Z INFO: Network log collection complete.
2026-11-03T20:17:56.624392701Z Wrote inspect data to must-gather.
2026-11-03T20:17:56.652422347Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:56.782352044Z INFO: Getting status from alertmanager-main-0
2026-11-03T20:17:56.880509701Z Wrote inspect data to must-gather.
2026-11-03T20:17:56.883117756Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:56.885697917Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-11-03T20:17:57.078793627Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:57.210180172Z INFO: Hosted cluster detected, get NTO container image from the TuneD pod
2026-11-03T20:17:57.292635075Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:57.297618302Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-11-03T20:17:57.365882347Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f60e56e0594610b09d7a64639223d0a6a314315bca9313914f95c902a1597c
2026-11-03T20:17:57.512680775Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:57.517496831Z daemonset.apps/perf-node-gather-daemonset created
2026-11-03T20:17:57.677137038Z Waiting for performance profile collector pods to become ready: 1
2026-11-03T20:17:57.749224156Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:57.751893162Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-11-03T20:17:57.988182229Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:58.061389100Z Gathering data for ns/openshift-network-console...
2026-11-03T20:17:58.213644423Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:58.216782721Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-11-03T20:17:58.455988484Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:58.522159654Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T20:17:58.554685685Z Gathering data for ns/openshift-console-operator...
2026-11-03T20:17:58.675756891Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:58.678523683Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-11-03T20:17:58.804805522Z Waiting for performance profile collector pods to become ready: 2
2026-11-03T20:17:58.871593224Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:58.999362640Z Gathering data for ns/openshift-console...
2026-11-03T20:17:59.082156381Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:59.085575363Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-11-03T20:17:59.263048043Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:59.472056385Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:59.474737852Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-11-03T20:17:59.583210588Z Gathering data for ns/openshift-cluster-storage-operator...
2026-11-03T20:17:59.658654807Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:59.856003045Z Wrote inspect data to /must-gather/istio.
2026-11-03T20:17:59.950283488Z Waiting for performance profile collector pods to become ready: 3
2026-11-03T20:18:00.178271155Z Gathering data for ns/openshift-dns-operator...
2026-11-03T20:18:00.567693481Z Gathering data for ns/openshift-dns...
2026-11-03T20:18:00.760394124Z Done executing Istio gather script
2026-11-03T20:18:01.080118669Z Waiting for performance profile collector pods to become ready: 4
2026-11-03T20:18:01.309983742Z Gathering data for ns/openshift-image-registry...
2026-11-03T20:18:01.834243670Z Gathering data for ns/openshift-ingress-operator...
2026-11-03T20:18:02.205350078Z Gathering data for ns/openshift-ingress...
2026-11-03T20:18:02.220535775Z Waiting for performance profile collector pods to become ready: 5
2026-11-03T20:18:02.592746509Z Gathering data for ns/openshift-ingress-canary...
2026-11-03T20:18:03.043179686Z Gathering data for ns/openshift-insights...
2026-11-03T20:18:03.334605689Z Waiting for performance profile collector pods to become ready: 6
2026-11-03T20:18:03.529062778Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T20:18:04.452226622Z Waiting for performance profile collector pods to become ready: 7
2026-11-03T20:18:05.583268448Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-11-03T20:18:05.835540877Z Gathering data for ns/kserve...
2026-11-03T20:18:05.865779794Z Collecting performance related data for node ip-10-0-140-232.us-east-2.compute.internal
2026-11-03T20:18:05.872345507Z Collecting performance related data for node ip-10-0-128-73.us-east-2.compute.internal
2026-11-03T20:18:05.879540152Z Collecting performance related data for node ip-10-0-140-51.us-east-2.compute.internal
2026-11-03T20:18:07.626272083Z Gathering data for ns/openshift-config...
2026-11-03T20:18:07.940152888Z Gathering data for ns/openshift-config-managed...
2026-11-03T20:18:08.492832116Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-11-03T20:18:08.535487024Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T20:18:08.869375488Z Gathering data for ns/openshift-kube-apiserver...
2026-11-03T20:18:09.223852879Z Gathering data for ns/openshift-kube-controller-manager...
2026-11-03T20:18:09.572700290Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-11-03T20:18:09.899203392Z Gathering data for ns/openshift-kube-scheduler...
2026-11-03T20:18:10.199802800Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-11-03T20:18:10.525414965Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-11-03T20:18:10.879499161Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-11-03T20:18:11.243861580Z Gathering data for ns/openshift-user-workload-monitoring...
2026-11-03T20:18:11.776513070Z Gathering data for ns/openshift-multus...
2026-11-03T20:18:13.072858641Z Collecting kubelet logs for node ip-10-0-140-232.us-east-2.compute.internal
2026-11-03T20:18:13.074611513Z Collecting kubelet logs for node ip-10-0-128-73.us-east-2.compute.internal
2026-11-03T20:18:13.076265386Z Collecting kubelet logs for node ip-10-0-140-51.us-east-2.compute.internal
2026-11-03T20:18:13.520208591Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-9v999 namespace
2026-11-03T20:18:13.525275491Z INFO: Node performance data collection complete.
2026-11-03T20:18:13.530937098Z Gathering data for ns/openshift-ovn-kubernetes...
2026-11-03T20:18:13.541132506Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T20:18:14.848667111Z Gathering data for ns/openshift-host-network...
2026-11-03T20:18:15.234576550Z Gathering data for ns/openshift-network-diagnostics...
2026-11-03T20:18:15.848971358Z Gathering data for ns/openshift-network-node-identity...
2026-11-03T20:18:16.329866362Z Gathering data for ns/openshift-network-operator...
2026-11-03T20:18:16.740271133Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-11-03T20:18:17.101970246Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-11-03T20:18:17.652677161Z Gathering data for ns/openshift-apiserver-operator...
2026-11-03T20:18:17.990399255Z Gathering data for ns/openshift-apiserver...
2026-11-03T20:18:18.487801923Z Gathering data for ns/openshift-controller-manager-operator...
2026-11-03T20:18:18.547908181Z [disk usage checker] Volume usage percentage: current = 9 ; allowed = 70
2026-11-03T20:18:18.771999927Z Gathering data for ns/openshift-controller-manager...
2026-11-03T20:18:19.100076608Z Gathering data for ns/openshift-cluster-samples-operator...
2026-11-03T20:18:19.795671635Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-11-03T20:18:20.114823165Z Gathering data for ns/openshift-service-ca-operator...
2026-11-03T20:18:20.474850521Z Gathering data for ns/openshift-service-ca...
2026-11-03T20:18:20.869716205Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-11-03T20:18:21.851079498Z Wrote inspect data to must-gather.
2026-11-03T20:18:21.851111724Z error: inspection completed with the errors occurred while gathering data:
2026-11-03T20:18:21.851111724Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-11-03T20:18:21.987326747Z error: the server doesn't have a resource type "clusters"
2026-11-03T20:18:22.107694908Z Caches written to disk