2026-07-07T16:53:30.746144559Z [disk usage checker] Started
2026-07-07T16:53:30.750428656Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:53:30.952513330Z Gathering data for ns/openshift-cluster-version...
2026-07-07T16:53:31.088902505Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T16:53:31.206449758Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T16:53:31.290040848Z Gathering data for ns/default...
2026-07-07T16:53:31.631139741Z Gathering data for ns/openshift...
2026-07-07T16:53:32.114582380Z Gathering data for ns/kube-system...
2026-07-07T16:53:32.869927035Z Gathering data for ns/openshift-etcd...
2026-07-07T16:53:33.205395516Z Wrote inspect data to must-gather.
2026-07-07T16:53:33.205430057Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T16:53:33.205430057Z     namespaces "assisted-installer" not found
2026-07-07T16:53:33.271582952Z Waiting on subprocesses to finish execution.
2026-07-07T16:53:33.276145275Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-07T16:53:33.280054467Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-07T16:53:33.280409198Z Executing Istio gather script
2026-07-07T16:53:33.282016099Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-07T16:53:33.288982525Z INFO: Gathering HAProxy config files
2026-07-07T16:53:33.352355090Z INFO: Collecting host service logs for crio
2026-07-07T16:53:33.352717794Z INFO: Collecting host service logs for kubelet
2026-07-07T16:53:33.353034041Z INFO: Collecting host service logs for rpm-ostreed
2026-07-07T16:53:33.353337414Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-07T16:53:33.353664836Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-07T16:53:33.353979518Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-07T16:53:33.354306761Z INFO: Collecting host service logs for NetworkManager
2026-07-07T16:53:33.354692251Z INFO: Collecting host service logs for openvswitch
2026-07-07T16:53:33.355048049Z INFO: Collecting host service logs for ovs-configuration
2026-07-07T16:53:33.355391009Z INFO: Collecting host service logs for ovsdb-server
2026-07-07T16:53:33.355716786Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-07T16:53:33.356084888Z INFO: Waiting for worker host service log collection to complete ...
2026-07-07T16:53:33.366767989Z INFO: Waiting for node performance related collection to complete ...
2026-07-07T16:53:33.556612801Z INFO: "metallb-operator" not detected. Skipping.
2026-07-07T16:53:33.657147936Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-07T16:53:33.804110718Z No resources found in openshift-etcd namespace.
2026-07-07T16:53:34.142603423Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-07T16:53:34.144059518Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-8bq88
2026-07-07T16:53:34.144113808Z insights-runtime-extractor-7wqld
2026-07-07T16:53:34.144135393Z insights-runtime-extractor-chnjv
2026-07-07T16:53:34.144155071Z insights-runtime-extractor-rj6pt
2026-07-07T16:53:34.216607248Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-07T16:53:34.288728830Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-07T16:53:34.402578182Z No resources found
2026-07-07T16:53:34.603724588Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-07T16:53:34.603821829Z INFO: on-disk MachineConfig config collection complete.
2026-07-07T16:53:34.653170755Z Wrote inspect data to must-gather.
2026-07-07T16:53:34.836638825Z error: the server doesn't have a resource type "performanceprofile"
2026-07-07T16:53:34.917667664Z ERROR: No running kube-apiserver pods found
2026-07-07T16:53:34.976717653Z INFO: Worker host service log collection to complete.
2026-07-07T16:53:34.992314638Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-07T16:53:35.175646549Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-07T16:53:35.175646549Z See 'oc rsync -h' for help and examples
2026-07-07T16:53:35.248932725Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-07T16:53:35.292010856Z Wrote inspect data to must-gather.
2026-07-07T16:53:35.302987183Z No resources found
2026-07-07T16:53:35.461354064Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T16:53:35.496040177Z Gathering data for ns/openshift-monitoring...
2026-07-07T16:53:35.646171688Z Wrote inspect data to must-gather.
2026-07-07T16:53:35.661438803Z Wrote inspect data to must-gather.
2026-07-07T16:53:35.715887384Z Wrote inspect data to must-gather.
2026-07-07T16:53:35.728753422Z INFO: 'previous-logs' folder not found on ip-10-0-128-52.ec2.internal, skipping...
2026-07-07T16:53:35.731387018Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-07T16:53:35.761714811Z Wrote inspect data to must-gather.
2026-07-07T16:53:35.770572637Z Wrote inspect data to must-gather.
2026-07-07T16:53:35.775699302Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:53:35.793697201Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-07T16:53:35.828760942Z INFO: Getting rules from prometheus-k8s-0
2026-07-07T16:53:35.932907521Z Wrote inspect data to must-gather.
2026-07-07T16:53:35.981076102Z error: the server doesn't have a resource type "machineconfigs"
2026-07-07T16:53:36.049600703Z tar: Removing leading `/' from member names
2026-07-07T16:53:36.055220275Z INFO: 'previous-logs' folder not found on ip-10-0-130-97.ec2.internal, skipping...
2026-07-07T16:53:36.060555364Z INFO: HAProxy config collection complete.
2026-07-07T16:53:36.132000662Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-07T16:53:36.205219865Z Wrote inspect data to must-gather.
2026-07-07T16:53:36.298462032Z INFO: Getting status/config from prometheus-k8s-0
2026-07-07T16:53:36.332883673Z INFO: 'previous-logs' folder not found on ip-10-0-133-178.ec2.internal, skipping...
2026-07-07T16:53:36.332994161Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-07T16:53:36.333041258Z INFO: Machine Config Daemon termination log collection complete.
2026-07-07T16:53:36.385059437Z Wrote inspect data to must-gather.
2026-07-07T16:53:36.465753062Z Wrote inspect data to must-gather.
2026-07-07T16:53:36.529375896Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-07T16:53:36.673930506Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-07T16:53:36.749000354Z Wrote inspect data to must-gather.
2026-07-07T16:53:36.757026311Z INFO: INTERCONNECT MODE
2026-07-07T16:53:36.757055454Z INFO: Gathering ovn-kubernetes DBs
2026-07-07T16:53:36.892337654Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-07T16:53:36.903762639Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-07T16:53:36.914030147Z INFO: Gathering OVN_Northbound from ovnkube-node-2nvdt...
2026-07-07T16:53:36.914353304Z INFO: Gathering OVN_Northbound from ovnkube-node-5fv7g...
2026-07-07T16:53:36.914654666Z INFO: Gathering OVN_Northbound from ovnkube-node-wkw66...
2026-07-07T16:53:36.914987162Z INFO: Gathering OVN_Southbound from ovnkube-node-2nvdt...
2026-07-07T16:53:36.915279721Z INFO: Gathering OVN_Southbound from ovnkube-node-5fv7g...
2026-07-07T16:53:36.915585510Z INFO: Gathering OVN_Southbound from ovnkube-node-wkw66...
2026-07-07T16:53:36.992561618Z Wrote inspect data to must-gather.
2026-07-07T16:53:37.216301236Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:37.228493686Z tar: Removing leading `/' from member names
2026-07-07T16:53:37.240519842Z tar: Removing leading `/' from member names
2026-07-07T16:53:37.250602907Z tar: Removing leading `/' from member names
2026-07-07T16:53:37.256967470Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-07T16:53:37.292783994Z tar: Removing leading `/' from member names
2026-07-07T16:53:37.294290752Z tar: Removing leading `/' from member names
2026-07-07T16:53:37.314893495Z tar: Removing leading `/' from member names
2026-07-07T16:53:37.340865942Z Wrote inspect data to must-gather.
2026-07-07T16:53:37.430874806Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-07T16:53:37.482089284Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:37.484941389Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-07T16:53:37.529609621Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-07T16:53:37.578740230Z Wrote inspect data to must-gather.
2026-07-07T16:53:37.722825149Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:37.759628962Z INFO: Waiting for network log collection to complete ...
2026-07-07T16:53:37.759679116Z INFO: Waiting for ovnk database copies to complete ...
2026-07-07T16:53:37.760152247Z INFO: Copying ovnk databases complete.
2026-07-07T16:53:37.761880690Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-07-07T16:53:37.764151477Z ovnk_database_store/
2026-07-07T16:53:37.764151477Z ovnk_database_store/ovnkube-node-5fv7g_nbdb
2026-07-07T16:53:37.770810165Z ovnk_database_store/ovnkube-node-2nvdt_nbdb
2026-07-07T16:53:37.778952377Z ovnk_database_store/ovnkube-node-wkw66_sbdb
2026-07-07T16:53:37.793288205Z Wrote inspect data to must-gather.
2026-07-07T16:53:37.795055944Z ovnk_database_store/ovnkube-node-wkw66_nbdb
2026-07-07T16:53:37.800714652Z ovnk_database_store/ovnkube-node-5fv7g_sbdb
2026-07-07T16:53:37.817805970Z ovnk_database_store/ovnkube-node-2nvdt_sbdb
2026-07-07T16:53:37.842041290Z INFO: Network log collection complete.
2026-07-07T16:53:37.942635133Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:37.947161442Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-07T16:53:37.992075301Z INFO: Getting status from alertmanager-main-0
2026-07-07T16:53:37.994499124Z W0707 16:53:37.994452    1533 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-07T16:53:38.000707395Z Wrote inspect data to must-gather.
2026-07-07T16:53:38.134922063Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-07T16:53:38.147350952Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:38.286058459Z Wrote inspect data to must-gather.
2026-07-07T16:53:38.321663592Z Gathering data for ns/openshift-network-console...
2026-07-07T16:53:38.333822145Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:38.336775250Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-07T16:53:38.534467936Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-07T16:53:38.537210298Z INFO: Fallback to identify the container image from release info
2026-07-07T16:53:38.555126435Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:38.743756616Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:38.745756728Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-07T16:53:38.798242272Z Gathering data for ns/openshift-console-operator...
2026-07-07T16:53:38.904401075Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:39.063179541Z Wrote inspect data to /must-gather/istio.
2026-07-07T16:53:39.253556046Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a
2026-07-07T16:53:39.331693125Z Gathering data for ns/openshift-console...
2026-07-07T16:53:39.379248598Z daemonset.apps/perf-node-gather-daemonset created
2026-07-07T16:53:39.517606176Z Waiting for performance profile collector pods to become ready: 1
2026-07-07T16:53:39.721758957Z Done executing Istio gather script
2026-07-07T16:53:39.958849551Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-07T16:53:40.472209793Z Gathering data for ns/openshift-dns-operator...
2026-07-07T16:53:40.620768470Z Waiting for performance profile collector pods to become ready: 2
2026-07-07T16:53:40.775056639Z Gathering data for ns/openshift-dns...
2026-07-07T16:53:40.781290521Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:53:41.404792639Z Gathering data for ns/openshift-image-registry...
2026-07-07T16:53:41.730819025Z Waiting for performance profile collector pods to become ready: 3
2026-07-07T16:53:41.892475515Z Gathering data for ns/openshift-ingress-operator...
2026-07-07T16:53:42.228673995Z Gathering data for ns/openshift-ingress...
2026-07-07T16:53:42.576509738Z Gathering data for ns/openshift-ingress-canary...
2026-07-07T16:53:42.827906356Z Waiting for performance profile collector pods to become ready: 4
2026-07-07T16:53:43.001671627Z Gathering data for ns/openshift-insights...
2026-07-07T16:53:43.937675315Z Waiting for performance profile collector pods to become ready: 5
2026-07-07T16:53:44.879505597Z Gathering data for ns/openshift-config...
2026-07-07T16:53:45.034439931Z Waiting for performance profile collector pods to become ready: 6
2026-07-07T16:53:45.153642936Z Gathering data for ns/openshift-config-managed...
2026-07-07T16:53:45.598684825Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-07T16:53:45.787242353Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:53:45.890850038Z Gathering data for ns/openshift-kube-apiserver...
2026-07-07T16:53:46.134914803Z Waiting for performance profile collector pods to become ready: 7
2026-07-07T16:53:46.250633900Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-07T16:53:46.597845320Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-07T16:53:46.907123104Z Gathering data for ns/openshift-kube-scheduler...
2026-07-07T16:53:47.214679449Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-07T16:53:47.254091982Z Waiting for performance profile collector pods to become ready: 8
2026-07-07T16:53:47.545970532Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-07T16:53:47.909985658Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-07T16:53:48.289863468Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-07T16:53:48.376933018Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-07T16:53:48.614490219Z Collecting performance related data for node ip-10-0-133-178.ec2.internal
2026-07-07T16:53:48.624159399Z Collecting performance related data for node ip-10-0-128-52.ec2.internal
2026-07-07T16:53:48.632746180Z Collecting performance related data for node ip-10-0-130-97.ec2.internal
2026-07-07T16:53:48.886885352Z Gathering data for ns/openshift-multus...
2026-07-07T16:53:50.497978635Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-07T16:53:50.791945654Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:53:51.854403895Z Gathering data for ns/openshift-host-network...
2026-07-07T16:53:52.196588536Z Gathering data for ns/openshift-network-diagnostics...
2026-07-07T16:53:52.745740343Z Gathering data for ns/openshift-network-node-identity...
2026-07-07T16:53:53.217021647Z Gathering data for ns/openshift-network-operator...
2026-07-07T16:53:53.646155288Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-07T16:53:53.941624105Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-07T16:53:54.492815711Z Gathering data for ns/openshift-apiserver-operator...
2026-07-07T16:53:54.825669774Z Gathering data for ns/openshift-apiserver...
2026-07-07T16:53:55.233038806Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-07T16:53:55.514254450Z Gathering data for ns/openshift-controller-manager...
2026-07-07T16:53:55.798899781Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-07T16:53:55.798963200Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-07-07T16:53:56.334177768Z Collecting kubelet logs for node ip-10-0-133-178.ec2.internal
2026-07-07T16:53:56.336454720Z Collecting kubelet logs for node ip-10-0-128-52.ec2.internal
2026-07-07T16:53:56.338711590Z Collecting kubelet logs for node ip-10-0-130-97.ec2.internal
2026-07-07T16:53:56.492801730Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-07T16:53:56.656777948Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-x7ktw namespace
2026-07-07T16:53:56.662201050Z INFO: Node performance data collection complete.
2026-07-07T16:53:56.854436427Z Gathering data for ns/openshift-service-ca-operator...
2026-07-07T16:53:57.165909886Z Gathering data for ns/openshift-service-ca...
2026-07-07T16:53:57.568213872Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-07T16:53:58.453355373Z Wrote inspect data to must-gather.
2026-07-07T16:53:58.453408532Z error: inspection completed with the errors occurred while gathering data:
2026-07-07T16:53:58.453408532Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-07T16:53:58.571393947Z error: the server doesn't have a resource type "clusters"
2026-07-07T16:53:58.809319434Z Caches written to disk