2026-09-05T03:09:38.851434850Z [disk usage checker] Started
2026-09-05T03:09:38.854737126Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-05T03:09:39.062889951Z Gathering data for ns/openshift-cluster-version...
2026-09-05T03:09:39.210816954Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-05T03:09:39.307310378Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-05T03:09:39.398097197Z Gathering data for ns/default...
2026-09-05T03:09:39.709965523Z Gathering data for ns/openshift...
2026-09-05T03:09:40.130596631Z Gathering data for ns/kube-system...
2026-09-05T03:09:40.887918543Z Gathering data for ns/openshift-etcd...
2026-09-05T03:09:41.217563613Z Wrote inspect data to must-gather.
2026-09-05T03:09:41.217596058Z error: inspection completed with the errors occurred while gathering data:
2026-09-05T03:09:41.217596058Z     namespaces "assisted-installer" not found
2026-09-05T03:09:41.697343820Z Waiting on subprocesses to finish execution.
2026-09-05T03:09:41.703403087Z INFO: Gathering HAProxy config files
2026-09-05T03:09:41.707263015Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-05T03:09:41.708609725Z Executing Istio gather script
2026-09-05T03:09:41.722206028Z INFO: Collecting host service logs for crio
2026-09-05T03:09:41.722484455Z INFO: Collecting host service logs for kubelet
2026-09-05T03:09:41.722793308Z INFO: Collecting host service logs for rpm-ostreed
2026-09-05T03:09:41.723171624Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-05T03:09:41.723513116Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-05T03:09:41.723850991Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-05T03:09:41.724199880Z INFO: Collecting host service logs for NetworkManager
2026-09-05T03:09:41.724538585Z INFO: Collecting host service logs for openvswitch
2026-09-05T03:09:41.724939733Z INFO: Collecting host service logs for ovs-configuration
2026-09-05T03:09:41.725390608Z INFO: Collecting host service logs for ovsdb-server
2026-09-05T03:09:41.725768146Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-05T03:09:41.726184597Z INFO: Waiting for worker host service log collection to complete ...
2026-09-05T03:09:41.729168890Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-05T03:09:41.730225854Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-05T03:09:41.882822406Z INFO: Waiting for node performance related collection to complete ...
2026-09-05T03:09:42.396406810Z No resources found in openshift-etcd namespace.
2026-09-05T03:09:42.570692086Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-05T03:09:42.719904838Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-05T03:09:42.773478953Z INFO: Collecting Insights Archives from insights-operator-76bcd6ccf5-92bgx
2026-09-05T03:09:42.773531703Z insights-runtime-extractor-b2fq5
2026-09-05T03:09:42.773551917Z insights-runtime-extractor-hp58v
2026-09-05T03:09:42.773571397Z insights-runtime-extractor-xvtld
2026-09-05T03:09:42.832504792Z No resources found
2026-09-05T03:09:42.937256094Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-05T03:09:42.977030527Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-05T03:09:43.074323068Z INFO: Worker host service log collection to complete.
2026-09-05T03:09:43.124589038Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-05T03:09:43.124874876Z INFO: on-disk MachineConfig config collection complete.
2026-09-05T03:09:43.433189102Z error: the server doesn't have a resource type "performanceprofile"
2026-09-05T03:09:43.518229200Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-05T03:09:43.550906840Z INFO: "metallb-operator" not detected. Skipping.
2026-09-05T03:09:43.633986341Z Wrote inspect data to must-gather.
2026-09-05T03:09:43.634544151Z No resources found
2026-09-05T03:09:43.641041316Z ERROR: No running kube-apiserver pods found
2026-09-05T03:09:43.672722786Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-05T03:09:43.838757724Z Wrote inspect data to must-gather.
2026-09-05T03:09:43.901551807Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-05T03:09:43.931708699Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-05T03:09:43.931708699Z See 'oc rsync -h' for help and examples
2026-09-05T03:09:43.967157720Z Wrote inspect data to must-gather.
2026-09-05T03:09:44.042888173Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-05T03:09:44.074338739Z Wrote inspect data to must-gather.
2026-09-05T03:09:44.079265839Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-05T03:09:44.106451241Z tar: INFO: HAProxy config collection complete.
2026-09-05T03:09:44.219874071Z Wrote inspect data to must-gather.
2026-09-05T03:09:44.245775835Z Wrote inspect data to must-gather.
2026-09-05T03:09:44.285710178Z INFO: 'previous-logs' folder not found on ip-10-0-128-63.ec2.internal, skipping...
2026-09-05T03:09:44.368530499Z Wrote inspect data to must-gather.
2026-09-05T03:09:44.403614824Z Gathering data for ns/openshift-monitoring...
2026-09-05T03:09:44.549505313Z Wrote inspect data to must-gather.
2026-09-05T03:09:44.604293561Z INFO: 'previous-logs' folder not found on ip-10-0-130-182.ec2.internal, skipping...
2026-09-05T03:09:44.614046273Z error: the server doesn't have a resource type "machineconfigs"
2026-09-05T03:09:44.708878881Z INFO: Getting rules from prometheus-k8s-0
2026-09-05T03:09:44.825885446Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-05T03:09:44.851814239Z Wrote inspect data to must-gather.
2026-09-05T03:09:44.922427087Z INFO: 'previous-logs' folder not found on ip-10-0-138-105.ec2.internal, skipping...
2026-09-05T03:09:44.922475209Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-05T03:09:44.922475209Z INFO: Machine Config Daemon termination log collection complete.
2026-09-05T03:09:45.092287524Z Wrote inspect data to must-gather.
2026-09-05T03:09:45.124158911Z Wrote inspect data to must-gather.
2026-09-05T03:09:45.217974825Z INFO: Getting status/config from prometheus-k8s-0
2026-09-05T03:09:45.225274673Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-05T03:09:45.265895136Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-05T03:09:45.444688495Z INFO: INTERCONNECT MODE
2026-09-05T03:09:45.444688495Z INFO: Gathering ovn-kubernetes DBs
2026-09-05T03:09:45.485611346Z Wrote inspect data to must-gather.
2026-09-05T03:09:45.609140057Z INFO: Gathering OVN_Northbound from ovnkube-node-7vnhf...
2026-09-05T03:09:45.609406826Z INFO: Gathering OVN_Northbound from ovnkube-node-8cc78...
2026-09-05T03:09:45.609678182Z INFO: Gathering OVN_Northbound from ovnkube-node-njtb5...
2026-09-05T03:09:45.610014679Z INFO: Gathering OVN_Southbound from ovnkube-node-7vnhf...
2026-09-05T03:09:45.610325993Z INFO: Gathering OVN_Southbound from ovnkube-node-8cc78...
2026-09-05T03:09:45.610648747Z INFO: Gathering OVN_Southbound from ovnkube-node-njtb5...
2026-09-05T03:09:45.764180376Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-05T03:09:45.776522977Z Wrote inspect data to must-gather.
2026-09-05T03:09:45.938605852Z tar: Removing leading `/' from member names
2026-09-05T03:09:45.939777790Z tar: Removing leading `/' from member names
2026-09-05T03:09:45.978126815Z tar: Removing leading `/' from member names
2026-09-05T03:09:45.997162232Z tar: Removing leading `/' from member names
2026-09-05T03:09:46.029177426Z tar: Removing leading `/' from member names
2026-09-05T03:09:46.047438127Z tar: Removing leading `/' from member names
2026-09-05T03:09:46.112014513Z Wrote inspect data to must-gather.
2026-09-05T03:09:46.131064727Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-05T03:09:46.233673555Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-05T03:09:46.374653688Z Wrote inspect data to must-gather.
2026-09-05T03:09:46.400473957Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-05T03:09:46.513288962Z INFO: Waiting for network log collection to complete ...
2026-09-05T03:09:46.513337952Z INFO: Waiting for ovnk database copies to complete ...
2026-09-05T03:09:46.513865348Z INFO: Copying ovnk databases complete.
2026-09-05T03:09:46.515681433Z 3.9M	must-gather/network_logs/ovnk_database_store
2026-09-05T03:09:46.518680316Z ovnk_database_store/
2026-09-05T03:09:46.518761689Z ovnk_database_store/ovnkube-node-njtb5_nbdb
2026-09-05T03:09:46.532630475Z ovnk_database_store/ovnkube-node-8cc78_sbdb
2026-09-05T03:09:46.558601508Z ovnk_database_store/ovnkube-node-8cc78_nbdb
2026-09-05T03:09:46.569338820Z ovnk_database_store/ovnkube-node-7vnhf_sbdb
2026-09-05T03:09:46.591867971Z ovnk_database_store/ovnkube-node-7vnhf_nbdb
2026-09-05T03:09:46.602021952Z ovnk_database_store/ovnkube-node-njtb5_sbdb
2026-09-05T03:09:46.618799611Z Wrote inspect data to must-gather.
2026-09-05T03:09:46.626930074Z INFO: Network log collection complete.
2026-09-05T03:09:46.658772834Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-05T03:09:46.680732669Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-05T03:09:46.833610924Z W0905 03:09:46.833553    1496 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-05T03:09:46.838479304Z Wrote inspect data to must-gather.
2026-09-05T03:09:46.853632101Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:46.875651433Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-05T03:09:47.102090226Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:47.104405045Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-05T03:09:47.104469191Z Gathering data for ns/openshift-network-console...
2026-09-05T03:09:47.114001437Z INFO: Getting status from alertmanager-main-0
2026-09-05T03:09:47.160474284Z Wrote inspect data to must-gather.
2026-09-05T03:09:47.323252602Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:47.467768845Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-05T03:09:47.469770605Z INFO: Fallback to identify the container image from release info
2026-09-05T03:09:47.542377234Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:47.544492862Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-05T03:09:47.623459271Z Gathering data for ns/openshift-console-operator...
2026-09-05T03:09:47.753760036Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:47.950541660Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:47.953492378Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-05T03:09:48.032125301Z Gathering data for ns/openshift-console...
2026-09-05T03:09:48.049636865Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f39c1daa89e19c5e987a3b759e646ffab6c1b2ab94a97ded81ddf13e73d0399
2026-09-05T03:09:48.207576711Z daemonset.apps/perf-node-gather-daemonset created
2026-09-05T03:09:48.214535205Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:48.345162221Z Waiting for performance profile collector pods to become ready: 1
2026-09-05T03:09:48.425720479Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:48.429102650Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-09-05T03:09:48.616174062Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-05T03:09:48.624962763Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:48.839689178Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:48.843167198Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-09-05T03:09:48.980607066Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-05T03:09:49.035133917Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:49.177514778Z Gathering data for ns/openshift-dns-operator...
2026-09-05T03:09:49.263156634Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:49.266019685Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-05T03:09:49.443869352Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:49.501147831Z Waiting for performance profile collector pods to become ready: 2
2026-09-05T03:09:49.522917823Z Gathering data for ns/openshift-dns...
2026-09-05T03:09:49.636754170Z Wrote inspect data to /must-gather/istio.
2026-09-05T03:09:50.167490115Z Gathering data for ns/openshift-image-registry...
2026-09-05T03:09:50.344039054Z Done executing Istio gather script
2026-09-05T03:09:50.624601241Z Waiting for performance profile collector pods to become ready: 3
2026-09-05T03:09:50.697461683Z Gathering data for ns/openshift-ingress-operator...
2026-09-05T03:09:51.063690637Z Gathering data for ns/openshift-ingress...
2026-09-05T03:09:51.411666492Z Gathering data for ns/openshift-ingress-canary...
2026-09-05T03:09:51.762314991Z Waiting for performance profile collector pods to become ready: 4
2026-09-05T03:09:51.823257138Z Gathering data for ns/openshift-insights...
2026-09-05T03:09:52.879734785Z Waiting for performance profile collector pods to become ready: 5
2026-09-05T03:09:53.985336838Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-05T03:09:54.009464513Z Waiting for performance profile collector pods to become ready: 6
2026-09-05T03:09:54.283874233Z Gathering data for ns/kserve...
2026-09-05T03:09:55.153653159Z Waiting for performance profile collector pods to become ready: 7
2026-09-05T03:09:55.938707212Z Gathering data for ns/openshift-config...
2026-09-05T03:09:56.277227313Z Gathering data for ns/openshift-config-managed...
2026-09-05T03:09:56.302860401Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-05T03:09:56.557476762Z Collecting performance related data for node ip-10-0-130-182.ec2.internal
2026-09-05T03:09:56.564611350Z Collecting performance related data for node ip-10-0-128-63.ec2.internal
2026-09-05T03:09:56.573280439Z Collecting performance related data for node ip-10-0-138-105.ec2.internal
2026-09-05T03:09:56.808345959Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-05T03:09:57.167242031Z Gathering data for ns/openshift-kube-apiserver...
2026-09-05T03:09:57.536992969Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-05T03:09:57.945492901Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-05T03:09:58.301236135Z Gathering data for ns/openshift-kube-scheduler...
2026-09-05T03:09:58.611029521Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-05T03:09:58.943180681Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-05T03:09:58.989971494Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-05T03:09:59.321993917Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-05T03:09:59.707007696Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-05T03:10:00.272281686Z Gathering data for ns/openshift-multus...
2026-09-05T03:10:01.851232126Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-05T03:10:03.134788898Z Gathering data for ns/openshift-host-network...
2026-09-05T03:10:03.466939180Z Gathering data for ns/openshift-network-diagnostics...
2026-09-05T03:10:03.998871273Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-05T03:10:04.045412989Z Gathering data for ns/openshift-network-node-identity...
2026-09-05T03:10:04.451288986Z Gathering data for ns/openshift-network-operator...
2026-09-05T03:10:04.871522853Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-05T03:10:05.018693067Z Collecting kubelet logs for node ip-10-0-130-182.ec2.internal
2026-09-05T03:10:05.020719109Z Collecting kubelet logs for node ip-10-0-128-63.ec2.internal
2026-09-05T03:10:05.022394565Z Collecting kubelet logs for node ip-10-0-138-105.ec2.internal
2026-09-05T03:10:05.198291634Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-05T03:10:05.412952167Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-zkkfn namespace
2026-09-05T03:10:05.417779182Z INFO: Node performance data collection complete.
2026-09-05T03:10:05.718021817Z Gathering data for ns/openshift-apiserver-operator...
2026-09-05T03:10:06.021575904Z Gathering data for ns/openshift-apiserver...
2026-09-05T03:10:06.422572078Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-05T03:10:06.707491949Z Gathering data for ns/openshift-controller-manager...
2026-09-05T03:10:06.982444002Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-05T03:10:07.618436039Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-05T03:10:07.899942081Z Gathering data for ns/openshift-service-ca-operator...
2026-09-05T03:10:08.227374357Z Gathering data for ns/openshift-service-ca...
2026-09-05T03:10:08.605177368Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-05T03:10:09.003565277Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-05T03:10:09.493619883Z Wrote inspect data to must-gather.
2026-09-05T03:10:09.493802237Z error: inspection completed with the errors occurred while gathering data:
2026-09-05T03:10:09.493802237Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-05T03:10:09.624688157Z error: the server doesn't have a resource type "clusters"
2026-09-05T03:10:09.770324690Z Caches written to disk