2026-07-29T15:23:40.555067657Z [disk usage checker] Started
2026-07-29T15:23:40.558330358Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T15:23:40.757586474Z Gathering data for ns/openshift-cluster-version...
2026-07-29T15:23:40.887567079Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T15:23:40.980928372Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T15:23:41.066724317Z Gathering data for ns/default...
2026-07-29T15:23:41.363515697Z Gathering data for ns/openshift...
2026-07-29T15:23:41.836917150Z Gathering data for ns/kube-system...
2026-07-29T15:23:42.581173147Z Gathering data for ns/openshift-etcd...
2026-07-29T15:23:42.958136002Z Wrote inspect data to must-gather.
2026-07-29T15:23:42.958169750Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T15:23:42.958169750Z     namespaces "assisted-installer" not found
2026-07-29T15:23:43.340353418Z Waiting on subprocesses to finish execution.
2026-07-29T15:23:43.343632618Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-29T15:23:43.351539928Z Executing Istio gather script
2026-07-29T15:23:43.354543833Z INFO: Gathering HAProxy config files
2026-07-29T15:23:43.366255667Z INFO: Collecting host service logs for crio
2026-07-29T15:23:43.366631214Z INFO: Collecting host service logs for kubelet
2026-07-29T15:23:43.367097096Z INFO: Collecting host service logs for rpm-ostreed
2026-07-29T15:23:43.367485303Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-29T15:23:43.367858142Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-29T15:23:43.368215048Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-29T15:23:43.368564181Z INFO: Collecting host service logs for NetworkManager
2026-07-29T15:23:43.368926527Z INFO: Collecting host service logs for openvswitch
2026-07-29T15:23:43.369265446Z INFO: Collecting host service logs for ovs-configuration
2026-07-29T15:23:43.369646294Z INFO: Collecting host service logs for ovsdb-server
2026-07-29T15:23:43.370006385Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-29T15:23:43.370420140Z INFO: Waiting for worker host service log collection to complete ...
2026-07-29T15:23:43.374955124Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-29T15:23:43.375251573Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-29T15:23:43.452215612Z INFO: Waiting for node performance related collection to complete ...
2026-07-29T15:23:43.885239089Z No resources found in openshift-etcd namespace.
2026-07-29T15:23:44.343839294Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-29T15:23:44.345624260Z No resources found
2026-07-29T15:23:44.432387604Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-29T15:23:44.450630187Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-mz4w8
2026-07-29T15:23:44.450677353Z insights-runtime-extractor-9k5fd
2026-07-29T15:23:44.450700134Z insights-runtime-extractor-bvlql
2026-07-29T15:23:44.450721035Z insights-runtime-extractor-kjkzb
2026-07-29T15:23:44.638159428Z error: the server doesn't have a resource type "performanceprofile"
2026-07-29T15:23:44.659035119Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-29T15:23:44.665833443Z Wrote inspect data to must-gather.
2026-07-29T15:23:44.684543241Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-29T15:23:44.692136857Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-29T15:23:44.766875509Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-29T15:23:44.766875509Z INFO: on-disk MachineConfig config collection complete.
2026-07-29T15:23:44.853097862Z INFO: Worker host service log collection to complete.
2026-07-29T15:23:44.934081251Z INFO: "metallb-operator" not detected. Skipping.
2026-07-29T15:23:44.972153173Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-29T15:23:44.972153173Z See 'oc rsync -h' for help and examples
2026-07-29T15:23:45.025283186Z Wrote inspect data to must-gather.
2026-07-29T15:23:45.131566118Z tar: Removing leading `/' from member names
2026-07-29T15:23:45.139513902Z INFO: HAProxy config collection complete.
2026-07-29T15:23:45.145144305Z ERROR: No running kube-apiserver pods found
2026-07-29T15:23:45.210102497Z Wrote inspect data to must-gather.
2026-07-29T15:23:45.474403047Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-29T15:23:45.610984319Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T15:23:45.697320287Z Gathering data for ns/openshift-monitoring...
2026-07-29T15:23:45.741478725Z No resources found
2026-07-29T15:23:45.789377742Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T15:23:45.822055400Z Wrote inspect data to must-gather.
2026-07-29T15:23:45.831810688Z Wrote inspect data to must-gather.
2026-07-29T15:23:45.947673829Z error: the server doesn't have a resource type "machineconfigs"
2026-07-29T15:23:45.970362078Z INFO: 'previous-logs' folder not found on ip-10-0-131-89.ec2.internal, skipping...
2026-07-29T15:23:45.980658895Z Wrote inspect data to must-gather.
2026-07-29T15:23:45.995416269Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-29T15:23:46.113647997Z Wrote inspect data to must-gather.
2026-07-29T15:23:46.178947829Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-29T15:23:46.276904302Z INFO: Getting rules from prometheus-k8s-0
2026-07-29T15:23:46.307740558Z INFO: 'previous-logs' folder not found on ip-10-0-135-6.ec2.internal, skipping...
2026-07-29T15:23:46.440705259Z Wrote inspect data to must-gather.
2026-07-29T15:23:46.459268835Z Wrote inspect data to must-gather.
2026-07-29T15:23:46.616516081Z INFO: 'previous-logs' folder not found on ip-10-0-140-209.ec2.internal, skipping...
2026-07-29T15:23:46.616636730Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-29T15:23:46.616692754Z INFO: Machine Config Daemon termination log collection complete.
2026-07-29T15:23:46.644567115Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-29T15:23:46.721736899Z Wrote inspect data to must-gather.
2026-07-29T15:23:46.759091079Z INFO: Getting status/config from prometheus-k8s-0
2026-07-29T15:23:46.914698373Z Wrote inspect data to must-gather.
2026-07-29T15:23:47.013513798Z Wrote inspect data to must-gather.
2026-07-29T15:23:47.136419020Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-29T15:23:47.181700984Z Wrote inspect data to must-gather.
2026-07-29T15:23:47.245331511Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-29T15:23:47.359137246Z INFO: INTERCONNECT MODE
2026-07-29T15:23:47.359200037Z INFO: Gathering ovn-kubernetes DBs
2026-07-29T15:23:47.448057715Z Wrote inspect data to must-gather.
2026-07-29T15:23:47.494348611Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-29T15:23:47.515412267Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-29T15:23:47.536156732Z INFO: Gathering OVN_Northbound from ovnkube-node-4kwwg...
2026-07-29T15:23:47.536412359Z INFO: Gathering OVN_Northbound from ovnkube-node-fknlj...
2026-07-29T15:23:47.536668905Z INFO: Gathering OVN_Northbound from ovnkube-node-vvr9v...
2026-07-29T15:23:47.536983820Z INFO: Gathering OVN_Southbound from ovnkube-node-4kwwg...
2026-07-29T15:23:47.537287666Z INFO: Gathering OVN_Southbound from ovnkube-node-fknlj...
2026-07-29T15:23:47.537579898Z INFO: Gathering OVN_Southbound from ovnkube-node-vvr9v...
2026-07-29T15:23:47.789818063Z Wrote inspect data to must-gather.
2026-07-29T15:23:47.877541476Z tar: Removing leading `/' from member names
2026-07-29T15:23:47.878237445Z tar: Removing leading `/' from member names
2026-07-29T15:23:47.891366440Z tar: Removing leading `/' from member names
2026-07-29T15:23:47.919211748Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-29T15:23:47.925133625Z tar: Removing leading `/' from member names
2026-07-29T15:23:47.934484813Z tar: Removing leading `/' from member names
2026-07-29T15:23:47.949342697Z tar: Removing leading `/' from member names
2026-07-29T15:23:47.951590698Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-29T15:23:48.072677814Z Wrote inspect data to must-gather.
2026-07-29T15:23:48.190571959Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:48.202188597Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-29T15:23:48.336239777Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-29T15:23:48.455575453Z Wrote inspect data to must-gather.
2026-07-29T15:23:48.464006171Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:48.466757188Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-29T15:23:48.523691581Z INFO: Waiting for network log collection to complete ...
2026-07-29T15:23:48.523740035Z INFO: Waiting for ovnk database copies to complete ...
2026-07-29T15:23:48.524215879Z INFO: Copying ovnk databases complete.
2026-07-29T15:23:48.526149976Z 2.8M	must-gather/network_logs/ovnk_database_store
2026-07-29T15:23:48.528463448Z ovnk_database_store/
2026-07-29T15:23:48.528463448Z ovnk_database_store/ovnkube-node-fknlj_sbdb
2026-07-29T15:23:48.548700084Z ovnk_database_store/ovnkube-node-fknlj_nbdb
2026-07-29T15:23:48.557437737Z ovnk_database_store/ovnkube-node-vvr9v_nbdb
2026-07-29T15:23:48.565072084Z ovnk_database_store/ovnkube-node-4kwwg_sbdb
2026-07-29T15:23:48.569363555Z Gathering data for ns/openshift-network-console...
2026-07-29T15:23:48.583624437Z ovnk_database_store/ovnkube-node-vvr9v_sbdb
2026-07-29T15:23:48.600200378Z ovnk_database_store/ovnkube-node-4kwwg_nbdb
2026-07-29T15:23:48.610197012Z INFO: Network log collection complete.
2026-07-29T15:23:48.702058147Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:48.733483302Z INFO: Getting status from alertmanager-main-0
2026-07-29T15:23:48.783601399Z Wrote inspect data to must-gather.
2026-07-29T15:23:48.910876710Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:48.914116811Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-29T15:23:49.088072302Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-29T15:23:49.088286188Z Gathering data for ns/openshift-console-operator...
2026-07-29T15:23:49.090039606Z INFO: Fallback to identify the container image from release info
2026-07-29T15:23:49.128701061Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:49.336455683Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:49.339144182Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-29T15:23:49.500712912Z Gathering data for ns/openshift-console...
2026-07-29T15:23:49.558444921Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:49.701794060Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-29T15:23:49.779564034Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:49.784210148Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-29T15:23:49.840128293Z daemonset.apps/perf-node-gather-daemonset created
2026-07-29T15:23:49.986291021Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:49.997382714Z Waiting for performance profile collector pods to become ready: 1
2026-07-29T15:23:50.088303603Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-29T15:23:50.198600543Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:50.201295189Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-29T15:23:50.392245817Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:50.601000808Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:50.603409442Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-29T15:23:50.620781508Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T15:23:50.667301479Z Gathering data for ns/openshift-dns-operator...
2026-07-29T15:23:50.786090388Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:50.989583552Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:50.994125216Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-29T15:23:51.009037678Z Gathering data for ns/openshift-dns...
2026-07-29T15:23:51.126899041Z Waiting for performance profile collector pods to become ready: 2
2026-07-29T15:23:51.160797535Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:51.336815534Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:51.340126967Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-29T15:23:51.513394877Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:51.725074136Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:51.727585135Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-29T15:23:51.757100203Z Gathering data for ns/openshift-image-registry...
2026-07-29T15:23:51.887602501Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:52.079754225Z Wrote inspect data to /must-gather/istio.
2026-07-29T15:23:52.278664720Z Waiting for performance profile collector pods to become ready: 3
2026-07-29T15:23:52.374214878Z Gathering data for ns/openshift-ingress-operator...
2026-07-29T15:23:52.767891879Z Gathering data for ns/openshift-ingress...
2026-07-29T15:23:52.783377192Z Done executing Istio gather script
2026-07-29T15:23:53.114171115Z Gathering data for ns/openshift-ingress-canary...
2026-07-29T15:23:53.394320045Z Waiting for performance profile collector pods to become ready: 4
2026-07-29T15:23:53.529963804Z Gathering data for ns/openshift-insights...
2026-07-29T15:23:54.504288932Z Waiting for performance profile collector pods to become ready: 5
2026-07-29T15:23:55.625828360Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T15:23:55.629557586Z Waiting for performance profile collector pods to become ready: 6
2026-07-29T15:23:55.849120853Z Gathering data for ns/kserve...
2026-07-29T15:23:56.745511287Z Waiting for performance profile collector pods to become ready: 7
2026-07-29T15:23:57.062741443Z Gathering data for ns/openshift-config...
2026-07-29T15:23:57.334581728Z Gathering data for ns/openshift-config-managed...
2026-07-29T15:23:57.762730339Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-29T15:23:57.878812548Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-29T15:23:58.054036345Z Gathering data for ns/openshift-kube-apiserver...
2026-07-29T15:23:58.165496443Z Collecting performance related data for node ip-10-0-135-6.ec2.internal
2026-07-29T15:23:58.172046043Z Collecting performance related data for node ip-10-0-131-89.ec2.internal
2026-07-29T15:23:58.178899480Z Collecting performance related data for node ip-10-0-140-209.ec2.internal
2026-07-29T15:23:58.380692079Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-29T15:23:58.727522291Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-29T15:23:59.110544506Z Gathering data for ns/openshift-kube-scheduler...
2026-07-29T15:23:59.442593346Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-29T15:23:59.877515373Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-29T15:24:00.204762090Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-29T15:24:00.572791036Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-29T15:24:00.633208890Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T15:24:01.195706184Z Gathering data for ns/openshift-multus...
2026-07-29T15:24:02.847474622Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-29T15:24:04.418972714Z Gathering data for ns/openshift-host-network...
2026-07-29T15:24:04.800816554Z Gathering data for ns/openshift-network-diagnostics...
2026-07-29T15:24:05.321701669Z Gathering data for ns/openshift-network-node-identity...
2026-07-29T15:24:05.638978374Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T15:24:05.767427933Z Gathering data for ns/openshift-network-operator...
2026-07-29T15:24:06.144394823Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-29T15:24:06.374753193Z Collecting kubelet logs for node ip-10-0-135-6.ec2.internal
2026-07-29T15:24:06.376439530Z Collecting kubelet logs for node ip-10-0-131-89.ec2.internal
2026-07-29T15:24:06.378211653Z Collecting kubelet logs for node ip-10-0-140-209.ec2.internal
2026-07-29T15:24:06.436404582Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-29T15:24:06.730156537Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-bvxxh namespace
2026-07-29T15:24:06.735820661Z INFO: Node performance data collection complete.
2026-07-29T15:24:06.971926314Z Gathering data for ns/openshift-apiserver-operator...
2026-07-29T15:24:07.309759028Z Gathering data for ns/openshift-apiserver...
2026-07-29T15:24:07.744839997Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-29T15:24:08.022072115Z Gathering data for ns/openshift-controller-manager...
2026-07-29T15:24:08.309071265Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-29T15:24:08.952650688Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-29T15:24:09.282690572Z Gathering data for ns/openshift-service-ca-operator...
2026-07-29T15:24:09.599168709Z Gathering data for ns/openshift-service-ca...
2026-07-29T15:24:09.963601386Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-29T15:24:10.643751682Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-29T15:24:10.823431988Z Wrote inspect data to must-gather.
2026-07-29T15:24:10.823485162Z error: inspection completed with the errors occurred while gathering data:
2026-07-29T15:24:10.823485162Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-29T15:24:10.951420183Z error: the server doesn't have a resource type "clusters"
2026-07-29T15:24:11.076788646Z Caches written to disk