2026-07-02T21:44:38.020942499Z [disk usage checker] Started
2026-07-02T21:44:38.024156358Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T21:44:38.235214864Z Gathering data for ns/openshift-cluster-version...
2026-07-02T21:44:38.376226951Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T21:44:38.470593487Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T21:44:38.553413197Z Gathering data for ns/default...
2026-07-02T21:44:38.854439041Z Gathering data for ns/openshift...
2026-07-02T21:44:39.251837772Z Gathering data for ns/kube-system...
2026-07-02T21:44:39.980646210Z Gathering data for ns/openshift-etcd...
2026-07-02T21:44:40.357164005Z Wrote inspect data to must-gather.
2026-07-02T21:44:40.357223126Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T21:44:40.357223126Z     namespaces "assisted-installer" not found
2026-07-02T21:44:40.733659348Z Waiting on subprocesses to finish execution.
2026-07-02T21:44:40.748622182Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-02T21:44:40.752535732Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-02T21:44:40.754278612Z Executing Istio gather script
2026-07-02T21:44:40.757070180Z INFO: Gathering HAProxy config files
2026-07-02T21:44:40.761933774Z INFO: Collecting host service logs for crio
2026-07-02T21:44:40.762288933Z INFO: Collecting host service logs for kubelet
2026-07-02T21:44:40.762617885Z INFO: Collecting host service logs for rpm-ostreed
2026-07-02T21:44:40.762911533Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-02T21:44:40.763232234Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-02T21:44:40.763355299Z INFO: Waiting for node performance related collection to complete ...
2026-07-02T21:44:40.763541866Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-02T21:44:40.763856628Z INFO: Collecting host service logs for NetworkManager
2026-07-02T21:44:40.764149996Z INFO: Collecting host service logs for openvswitch
2026-07-02T21:44:40.764498961Z INFO: Collecting host service logs for ovs-configuration
2026-07-02T21:44:40.764803488Z INFO: Collecting host service logs for ovsdb-server
2026-07-02T21:44:40.765099113Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-02T21:44:40.765481760Z INFO: Waiting for worker host service log collection to complete ...
2026-07-02T21:44:40.774426491Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-02T21:44:41.268758064Z No resources found in openshift-etcd namespace.
2026-07-02T21:44:41.441591352Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-bwl8z
2026-07-02T21:44:41.441632472Z insights-runtime-extractor-gsgsx
2026-07-02T21:44:41.441650971Z insights-runtime-extractor-m6bc9
2026-07-02T21:44:41.441668991Z insights-runtime-extractor-pvf5d
2026-07-02T21:44:41.541529714Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-02T21:44:41.559322419Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-02T21:44:41.559353308Z INFO: on-disk MachineConfig config collection complete.
2026-07-02T21:44:41.700041760Z No resources found
2026-07-02T21:44:41.778003753Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-02T21:44:41.784312854Z error: the server doesn't have a resource type "performanceprofile"
2026-07-02T21:44:41.895447911Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-02T21:44:42.070853812Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-02T21:44:42.213611348Z INFO: "metallb-operator" not detected. Skipping.
2026-07-02T21:44:42.408246726Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-02T21:44:42.429841509Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-02T21:44:42.429841509Z See 'oc rsync -h' for help and examples
2026-07-02T21:44:42.447833540Z INFO: Worker host service log collection to complete.
2026-07-02T21:44:42.665657873Z Wrote inspect data to must-gather.
2026-07-02T21:44:42.755020409Z Wrote inspect data to must-gather.
2026-07-02T21:44:42.926477987Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-02T21:44:42.926522761Z Wrote inspect data to must-gather.
2026-07-02T21:44:42.928899693Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T21:44:42.997522767Z tar: Removing leading `/' from member names
2026-07-02T21:44:43.008956479Z INFO: HAProxy config collection complete.
2026-07-02T21:44:43.095047880Z INFO: 'previous-logs' folder not found on ip-10-0-133-168.ec2.internal, skipping...
2026-07-02T21:44:43.105377072Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T21:44:43.135825072Z Gathering data for ns/openshift-monitoring...
2026-07-02T21:44:43.201543698Z ERROR: No running kube-apiserver pods found
2026-07-02T21:44:43.373085410Z error: the server doesn't have a resource type "machineconfigs"
2026-07-02T21:44:43.385255753Z No resources found
2026-07-02T21:44:43.508744635Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-02T21:44:43.554328094Z Wrote inspect data to must-gather.
2026-07-02T21:44:43.617428513Z Wrote inspect data to must-gather.
2026-07-02T21:44:43.626833389Z Wrote inspect data to must-gather.
2026-07-02T21:44:43.654421231Z Wrote inspect data to must-gather.
2026-07-02T21:44:43.731052521Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-02T21:44:43.780261106Z INFO: 'previous-logs' folder not found on ip-10-0-135-227.ec2.internal, skipping...
2026-07-02T21:44:43.868928566Z INFO: Getting rules from prometheus-k8s-0
2026-07-02T21:44:43.988263929Z Wrote inspect data to must-gather.
2026-07-02T21:44:44.029911389Z Wrote inspect data to must-gather.
2026-07-02T21:44:44.091411543Z INFO: 'previous-logs' folder not found on ip-10-0-137-60.ec2.internal, skipping...
2026-07-02T21:44:44.091471648Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-02T21:44:44.091552131Z INFO: Machine Config Daemon termination log collection complete.
2026-07-02T21:44:44.216655813Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-02T21:44:44.305043799Z Wrote inspect data to must-gather.
2026-07-02T21:44:44.347014988Z INFO: Getting status/config from prometheus-k8s-0
2026-07-02T21:44:44.462149787Z Wrote inspect data to must-gather.
2026-07-02T21:44:44.475310793Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-02T21:44:44.571006733Z Wrote inspect data to must-gather.
2026-07-02T21:44:44.710832895Z Wrote inspect data to must-gather.
2026-07-02T21:44:44.823695670Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-02T21:44:44.845850117Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-02T21:44:44.946883365Z INFO: INTERCONNECT MODE
2026-07-02T21:44:44.946921638Z INFO: Gathering ovn-kubernetes DBs
2026-07-02T21:44:44.965583860Z Wrote inspect data to must-gather.
2026-07-02T21:44:45.093210109Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-02T21:44:45.123802184Z INFO: Gathering OVN_Northbound from ovnkube-node-2725k...
2026-07-02T21:44:45.124043932Z INFO: Gathering OVN_Northbound from ovnkube-node-9gz4j...
2026-07-02T21:44:45.124344938Z INFO: Gathering OVN_Northbound from ovnkube-node-vtvxg...
2026-07-02T21:44:45.124660916Z INFO: Gathering OVN_Southbound from ovnkube-node-2725k...
2026-07-02T21:44:45.124934723Z INFO: Gathering OVN_Southbound from ovnkube-node-9gz4j...
2026-07-02T21:44:45.125434634Z INFO: Gathering OVN_Southbound from ovnkube-node-vtvxg...
2026-07-02T21:44:45.238916859Z Wrote inspect data to must-gather.
2026-07-02T21:44:45.452480641Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-02T21:44:45.463525311Z tar: Removing leading `/' from member names
2026-07-02T21:44:45.473796727Z tar: Removing leading `/' from member names
2026-07-02T21:44:45.498475822Z tar: Removing leading `/' from member names
2026-07-02T21:44:45.498773121Z tar: Removing leading `/' from member names
2026-07-02T21:44:45.521305660Z tar: Removing leading `/' from member names
2026-07-02T21:44:45.552899439Z tar: Removing leading `/' from member names
2026-07-02T21:44:45.621254930Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-02T21:44:45.634350617Z W0702 21:44:45.634303    1386 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-02T21:44:45.641046508Z Wrote inspect data to must-gather.
2026-07-02T21:44:45.660346860Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-02T21:44:45.774202856Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-02T21:44:45.874796464Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:45.897011841Z Gathering data for ns/openshift-network-console...
2026-07-02T21:44:46.029745240Z Wrote inspect data to must-gather.
2026-07-02T21:44:46.136322587Z INFO: Waiting for network log collection to complete ...
2026-07-02T21:44:46.136357141Z INFO: Waiting for ovnk database copies to complete ...
2026-07-02T21:44:46.136831917Z INFO: Copying ovnk databases complete.
2026-07-02T21:44:46.138665968Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-02T21:44:46.141479825Z ovnk_database_store/
2026-07-02T21:44:46.141529790Z ovnk_database_store/ovnkube-node-9gz4j_nbdb
2026-07-02T21:44:46.145439149Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:46.149804615Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-02T21:44:46.155324209Z ovnk_database_store/ovnkube-node-2725k_sbdb
2026-07-02T21:44:46.182363423Z ovnk_database_store/ovnkube-node-vtvxg_sbdb
2026-07-02T21:44:46.206328711Z ovnk_database_store/ovnkube-node-vtvxg_nbdb
2026-07-02T21:44:46.215115345Z ovnk_database_store/ovnkube-node-2725k_nbdb
2026-07-02T21:44:46.224906038Z ovnk_database_store/ovnkube-node-9gz4j_sbdb
2026-07-02T21:44:46.250920356Z INFO: Network log collection complete.
2026-07-02T21:44:46.268854947Z INFO: Getting status from alertmanager-main-0
2026-07-02T21:44:46.386772594Z Wrote inspect data to must-gather.
2026-07-02T21:44:46.403262560Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:46.428335520Z Gathering data for ns/openshift-console-operator...
2026-07-02T21:44:46.624543476Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:46.628297337Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-02T21:44:46.680947819Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-02T21:44:46.683798817Z INFO: Fallback to identify the container image from release info
2026-07-02T21:44:46.838873219Z Gathering data for ns/openshift-console...
2026-07-02T21:44:46.852584673Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:47.048397758Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:47.050848468Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-02T21:44:47.294138403Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:47.320995005Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-02T21:44:47.384198068Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-02T21:44:47.494019074Z daemonset.apps/perf-node-gather-daemonset created
2026-07-02T21:44:47.552227434Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:47.555336994Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-02T21:44:47.645251895Z Waiting for performance profile collector pods to become ready: 1
2026-07-02T21:44:47.750839286Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:47.980693264Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:47.982758976Z Gathering data for ns/openshift-dns-operator...
2026-07-02T21:44:47.984092302Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-02T21:44:48.110387878Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T21:44:48.179603886Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:48.311334473Z Gathering data for ns/openshift-dns...
2026-07-02T21:44:48.374882147Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:48.377060388Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-02T21:44:48.533732069Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:48.748481683Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:48.751871461Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-02T21:44:48.790246183Z Waiting for performance profile collector pods to become ready: 2
2026-07-02T21:44:48.939095545Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:48.945387808Z Gathering data for ns/openshift-image-registry...
2026-07-02T21:44:49.133301587Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:49.135583320Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-02T21:44:49.303567996Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:49.436347693Z Gathering data for ns/openshift-ingress-operator...
2026-07-02T21:44:49.521786980Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:49.525609074Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-02T21:44:49.683010902Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:49.822046964Z Gathering data for ns/openshift-ingress...
2026-07-02T21:44:49.895241426Z Wrote inspect data to /must-gather/istio.
2026-07-02T21:44:49.946686622Z Waiting for performance profile collector pods to become ready: 3
2026-07-02T21:44:50.190264423Z Gathering data for ns/openshift-ingress-canary...
2026-07-02T21:44:50.593753044Z Gathering data for ns/openshift-insights...
2026-07-02T21:44:50.598907771Z Done executing Istio gather script
2026-07-02T21:44:51.078071744Z Waiting for performance profile collector pods to become ready: 4
2026-07-02T21:44:52.199593637Z Waiting for performance profile collector pods to become ready: 5
2026-07-02T21:44:52.795950714Z Gathering data for ns/kserve...
2026-07-02T21:44:53.114706990Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T21:44:53.318522837Z Waiting for performance profile collector pods to become ready: 6
2026-07-02T21:44:54.343487915Z Gathering data for ns/openshift-config...
2026-07-02T21:44:54.449096091Z Waiting for performance profile collector pods to become ready: 7
2026-07-02T21:44:54.621437175Z Gathering data for ns/openshift-config-managed...
2026-07-02T21:44:55.037218291Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-02T21:44:55.289658133Z Gathering data for ns/openshift-kube-apiserver...
2026-07-02T21:44:55.577386136Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-02T21:44:55.608362170Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-02T21:44:55.844240205Z Collecting performance related data for node ip-10-0-137-60.ec2.internal
2026-07-02T21:44:55.852266007Z Collecting performance related data for node ip-10-0-133-168.ec2.internal
2026-07-02T21:44:55.863126485Z Collecting performance related data for node ip-10-0-135-227.ec2.internal
2026-07-02T21:44:55.947713570Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-02T21:44:56.287455611Z Gathering data for ns/openshift-kube-scheduler...
2026-07-02T21:44:56.626028123Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-02T21:44:57.081719181Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-02T21:44:57.504970504Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-02T21:44:57.810111061Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-02T21:44:58.119009463Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T21:44:58.311670680Z Gathering data for ns/openshift-multus...
2026-07-02T21:44:59.703680368Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-02T21:45:00.946849699Z Gathering data for ns/openshift-host-network...
2026-07-02T21:45:01.252632183Z Gathering data for ns/openshift-network-diagnostics...
2026-07-02T21:45:01.780005466Z Gathering data for ns/openshift-network-node-identity...
2026-07-02T21:45:02.179527349Z Gathering data for ns/openshift-network-operator...
2026-07-02T21:45:02.554947242Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-02T21:45:02.803951715Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-02T21:45:03.126081318Z [disk usage checker] Volume usage percentage: current = 14 ; allowed = 70
2026-07-02T21:45:03.312358332Z Gathering data for ns/openshift-apiserver-operator...
2026-07-02T21:45:03.612350456Z Gathering data for ns/openshift-apiserver...
2026-07-02T21:45:03.871319211Z Collecting kubelet logs for node ip-10-0-137-60.ec2.internal
2026-07-02T21:45:03.873309322Z Collecting kubelet logs for node ip-10-0-133-168.ec2.internal
2026-07-02T21:45:03.875090373Z Collecting kubelet logs for node ip-10-0-135-227.ec2.internal
2026-07-02T21:45:04.065125957Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-02T21:45:04.280815348Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-thbg4 namespace
2026-07-02T21:45:04.286008888Z INFO: Node performance data collection complete.
2026-07-02T21:45:04.382370245Z Gathering data for ns/openshift-controller-manager...
2026-07-02T21:45:04.681552372Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-02T21:45:05.298889919Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-02T21:45:05.577375364Z Gathering data for ns/openshift-service-ca-operator...
2026-07-02T21:45:05.884602276Z Gathering data for ns/openshift-service-ca...
2026-07-02T21:45:06.222263906Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-02T21:45:07.091626725Z Wrote inspect data to must-gather.
2026-07-02T21:45:07.091666961Z error: inspection completed with the errors occurred while gathering data:
2026-07-02T21:45:07.091666961Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-02T21:45:07.217331524Z error: the server doesn't have a resource type "clusters"
2026-07-02T21:45:07.357979819Z Caches written to disk