2026-07-02T14:13:05.562257741Z [disk usage checker] Started
2026-07-02T14:13:05.569249142Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-02T14:13:05.766331097Z Gathering data for ns/openshift-cluster-version...
2026-07-02T14:13:05.916975207Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:13:06.020228413Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:13:06.107569019Z Gathering data for ns/default...
2026-07-02T14:13:06.488045247Z Gathering data for ns/openshift...
2026-07-02T14:13:06.941128235Z Gathering data for ns/kube-system...
2026-07-02T14:13:07.785032281Z Gathering data for ns/openshift-etcd...
2026-07-02T14:13:08.142548865Z Wrote inspect data to must-gather.
2026-07-02T14:13:08.142583813Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:13:08.142583813Z     namespaces "assisted-installer" not found
2026-07-02T14:13:08.472023121Z Waiting on subprocesses to finish execution.
2026-07-02T14:13:08.477143723Z Executing Istio gather script
2026-07-02T14:13:08.481039174Z INFO: Collecting host service logs for crio
2026-07-02T14:13:08.481334273Z INFO: Collecting host service logs for kubelet
2026-07-02T14:13:08.481720639Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T14:13:08.482052647Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T14:13:08.482102094Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T14:13:08.482382773Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T14:13:08.482870076Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T14:13:08.483225854Z INFO: Collecting host service logs for NetworkManager
2026-07-02T14:13:08.483577334Z INFO: Collecting host service logs for openvswitch
2026-07-02T14:13:08.483921299Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T14:13:08.484262526Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T14:13:08.484614236Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T14:13:08.485018702Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T14:13:08.485231165Z INFO: Gathering HAProxy config files
2026-07-02T14:13:08.487452880Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T14:13:08.496041881Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T14:13:08.790412632Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T14:13:09.198818555Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T14:13:09.198923244Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T14:13:09.248200703Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T14:13:09.400808562Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T14:13:09.547834902Z Wrote inspect data to must-gather.
2026-07-02T14:13:09.811189854Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T14:13:09.851318656Z No resources found in openshift-etcd namespace.
2026-07-02T14:13:09.879162921Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T14:13:09.883451903Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-z4sj4
2026-07-02T14:13:09.883524994Z insights-runtime-extractor-9brbm
2026-07-02T14:13:09.883561223Z insights-runtime-extractor-n7k8v
2026-07-02T14:13:09.883579741Z insights-runtime-extractor-nxgcc
2026-07-02T14:13:10.045683922Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T14:13:10.053625177Z ERROR: No running kube-apiserver pods found
2026-07-02T14:13:10.069677935Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T14:13:10.079072625Z INFO: Worker host service log collection to complete.
2026-07-02T14:13:10.299954083Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T14:13:10.345174449Z Wrote inspect data to must-gather.
2026-07-02T14:13:10.350604061Z No resources found
2026-07-02T14:13:10.425173039Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T14:13:10.425173039Z See 'oc rsync -h' for help and examples
2026-07-02T14:13:10.542717368Z No resources found
2026-07-02T14:13:10.556098065Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T14:13:10.653917735Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-02T14:13:10.783545642Z Wrote inspect data to must-gather.
2026-07-02T14:13:10.829271801Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:13:11.000027234Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T14:13:11.001525202Z Wrote inspect data to must-gather.
2026-07-02T14:13:11.066261227Z INFO: 'previous-logs' folder not found on ip-10-0-133-48.ec2.internal, skipping...
2026-07-02T14:13:11.070746728Z Wrote inspect data to must-gather.
2026-07-02T14:13:11.175829197Z Wrote inspect data to must-gather.
2026-07-02T14:13:11.204516203Z tar: Removing leading `/' from member names
2026-07-02T14:13:11.210049788Z INFO: HAProxy config collection complete.
2026-07-02T14:13:11.214207902Z Wrote inspect data to must-gather.
2026-07-02T14:13:11.352875698Z Gathering data for ns/openshift-monitoring...
2026-07-02T14:13:11.433752249Z INFO: 'previous-logs' folder not found on ip-10-0-136-90.ec2.internal, skipping...
2026-07-02T14:13:11.441691674Z Wrote inspect data to must-gather.
2026-07-02T14:13:11.465602164Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T14:13:11.619394171Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T14:13:11.683698172Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T14:13:11.751921139Z INFO: 'previous-logs' folder not found on ip-10-0-137-226.ec2.internal, skipping...
2026-07-02T14:13:11.751958978Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T14:13:11.751981811Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T14:13:11.779494070Z Wrote inspect data to must-gather.
2026-07-02T14:13:11.836160307Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T14:13:11.927859687Z Wrote inspect data to must-gather.
2026-07-02T14:13:12.040009706Z Wrote inspect data to must-gather.
2026-07-02T14:13:12.084227149Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T14:13:12.102506857Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T14:13:12.327378058Z Wrote inspect data to must-gather.
2026-07-02T14:13:12.359955456Z INFO: INTERCONNECT MODE
2026-07-02T14:13:12.359993582Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T14:13:12.510912334Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T14:13:12.534461908Z INFO: Gathering OVN_Northbound from ovnkube-node-825xg...
2026-07-02T14:13:12.534834425Z INFO: Gathering OVN_Northbound from ovnkube-node-lwl25...
2026-07-02T14:13:12.535123679Z INFO: Gathering OVN_Northbound from ovnkube-node-xtms2...
2026-07-02T14:13:12.535456843Z INFO: Gathering OVN_Southbound from ovnkube-node-825xg...
2026-07-02T14:13:12.535765404Z INFO: Gathering OVN_Southbound from ovnkube-node-lwl25...
2026-07-02T14:13:12.536041872Z INFO: Gathering OVN_Southbound from ovnkube-node-xtms2...
2026-07-02T14:13:12.621320222Z Wrote inspect data to must-gather.
2026-07-02T14:13:12.861447266Z tar: Removing leading `/' from member names
2026-07-02T14:13:12.877975035Z tar: Removing leading `/' from member names
2026-07-02T14:13:12.880603822Z tar: Removing leading `/' from member names
2026-07-02T14:13:12.886454270Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T14:13:12.888702070Z tar: Removing leading `/' from member names
2026-07-02T14:13:12.895756853Z tar: Removing leading `/' from member names
2026-07-02T14:13:12.919101157Z tar: Removing leading `/' from member names
2026-07-02T14:13:12.990368127Z Wrote inspect data to must-gather.
2026-07-02T14:13:13.100336130Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T14:13:13.193214828Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T14:13:13.249096975Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T14:13:13.260858051Z Wrote inspect data to must-gather.
2026-07-02T14:13:13.485699622Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:13.496782792Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T14:13:13.501646139Z Wrote inspect data to must-gather.
2026-07-02T14:13:13.507992561Z INFO: Waiting for network log collection to complete ...
2026-07-02T14:13:13.508034960Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T14:13:13.508564800Z INFO: Copying ovnk databases complete.
2026-07-02T14:13:13.511020848Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-02T14:13:13.513930484Z ovnk_database_store/
2026-07-02T14:13:13.513930484Z ovnk_database_store/ovnkube-node-825xg_sbdb
2026-07-02T14:13:13.536701108Z ovnk_database_store/ovnkube-node-825xg_nbdb
2026-07-02T14:13:13.545857390Z ovnk_database_store/ovnkube-node-xtms2_sbdb
2026-07-02T14:13:13.568991414Z ovnk_database_store/ovnkube-node-xtms2_nbdb
2026-07-02T14:13:13.579522176Z ovnk_database_store/ovnkube-node-lwl25_sbdb
2026-07-02T14:13:13.602993232Z ovnk_database_store/ovnkube-node-lwl25_nbdb
2026-07-02T14:13:13.617582374Z INFO: Network log collection complete.
2026-07-02T14:13:13.728753406Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:13.731296398Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T14:13:13.748977264Z W0702 14:13:13.748928    1469 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-02T14:13:13.753868399Z Wrote inspect data to must-gather.
2026-07-02T14:13:13.778187492Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T14:13:13.954170761Z INFO: Getting status from alertmanager-main-0
2026-07-02T14:13:13.959962307Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:13.997787977Z Gathering data for ns/openshift-network-console...
2026-07-02T14:13:14.112808818Z Wrote inspect data to must-gather.
2026-07-02T14:13:14.203397805Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:14.205823178Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T14:13:14.410804584Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T14:13:14.414906042Z INFO: Fallback to identify the container image from release info
2026-07-02T14:13:14.428266831Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:14.513021385Z Gathering data for ns/openshift-console-operator...
2026-07-02T14:13:14.652861142Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:14.655178897Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T14:13:14.877534835Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:14.933730918Z Gathering data for ns/openshift-console...
2026-07-02T14:13:15.093077805Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:15.095368265Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-02T14:13:15.266418319Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:15.509454920Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:15.512319242Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T14:13:15.512896052Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-02T14:13:15.521445741Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f
2026-07-02T14:13:15.671636047Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T14:13:15.702457036Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:15.731366590Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-02T14:13:15.822752303Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T14:13:15.942783538Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:15.945533024Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-02T14:13:16.158333421Z Gathering data for ns/openshift-dns-operator...
2026-07-02T14:13:16.164778769Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:16.398621938Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:16.402822192Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-02T14:13:16.518206909Z Gathering data for ns/openshift-dns...
2026-07-02T14:13:16.588937657Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:16.774016597Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:16.776504829Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-02T14:13:16.979401496Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T14:13:16.994792368Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:17.192173412Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:17.195357885Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T14:13:17.349935575Z Gathering data for ns/openshift-image-registry...
2026-07-02T14:13:17.388511981Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:17.587963460Z Wrote inspect data to /must-gather/istio.
2026-07-02T14:13:17.896224525Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T14:13:18.103705505Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T14:13:18.302189083Z Gathering data for ns/openshift-ingress...
2026-07-02T14:13:18.338881485Z Done executing Istio gather script
2026-07-02T14:13:18.651863079Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T14:13:19.075213101Z Gathering data for ns/openshift-insights...
2026-07-02T14:13:19.222409031Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T14:13:20.349712220Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T14:13:20.737462127Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-02T14:13:21.475630942Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T14:13:21.486132173Z Gathering data for ns/kserve...
2026-07-02T14:13:22.615793380Z Waiting for performance profile collector pods to become ready: 7
2026-07-02T14:13:23.290856544Z Gathering data for ns/openshift-config...
2026-07-02T14:13:23.611806814Z Gathering data for ns/openshift-config-managed...
2026-07-02T14:13:23.754918530Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T14:13:24.027407234Z Collecting performance related data for node ip-10-0-137-226.ec2.internal
2026-07-02T14:13:24.036921125Z Collecting performance related data for node ip-10-0-136-90.ec2.internal
2026-07-02T14:13:24.045263200Z Collecting performance related data for node ip-10-0-133-48.ec2.internal
2026-07-02T14:13:24.112754575Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T14:13:24.494988587Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T14:13:24.857066427Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T14:13:25.258958863Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T14:13:25.671937842Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T14:13:25.748225457Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-02T14:13:26.002036373Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T14:13:26.334684109Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T14:13:26.761331179Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T14:13:27.199812806Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T14:13:27.786548889Z Gathering data for ns/openshift-multus...
2026-07-02T14:13:29.471091376Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T14:13:30.754591675Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-02T14:13:30.881773807Z Gathering data for ns/openshift-host-network...
2026-07-02T14:13:31.271246194Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T14:13:31.933024202Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T14:13:32.408201640Z Gathering data for ns/openshift-network-operator...
2026-07-02T14:13:32.663959272Z Collecting kubelet logs for node ip-10-0-137-226.ec2.internal
2026-07-02T14:13:32.665920115Z Collecting kubelet logs for node ip-10-0-136-90.ec2.internal
2026-07-02T14:13:32.667811786Z Collecting kubelet logs for node ip-10-0-133-48.ec2.internal
2026-07-02T14:13:32.948547269Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T14:13:33.090128935Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-hs76s namespace
2026-07-02T14:13:33.097602273Z INFO: Node performance data collection complete.
2026-07-02T14:13:33.303372223Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T14:13:33.916384637Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T14:13:34.236875909Z Gathering data for ns/openshift-apiserver...
2026-07-02T14:13:34.660502694Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T14:13:34.958425560Z Gathering data for ns/openshift-controller-manager...
2026-07-02T14:13:35.298041434Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T14:13:35.759264839Z [disk usage checker] Volume usage percentage: current = 24 ; allowed = 70
2026-07-02T14:13:36.004147116Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T14:13:36.304545989Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T14:13:36.636218673Z Gathering data for ns/openshift-service-ca...
2026-07-02T14:13:37.026984717Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T14:13:38.103740239Z Wrote inspect data to must-gather.
2026-07-02T14:13:38.103785699Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T14:13:38.103785699Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T14:13:38.237712967Z error: the server doesn't have a resource type "clusters"
2026-07-02T14:13:38.377317107Z Caches written to disk