2026-07-21T21:40:41.167400423Z [disk usage checker] Started
2026-07-21T21:40:41.170692212Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T21:40:41.371003560Z Gathering data for ns/openshift-cluster-version...
2026-07-21T21:40:41.530896321Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-21T21:40:41.639733106Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T21:40:41.721793525Z Gathering data for ns/default...
2026-07-21T21:40:42.032820890Z Gathering data for ns/openshift...
2026-07-21T21:40:42.463282165Z Gathering data for ns/kube-system...
2026-07-21T21:40:43.330071561Z Gathering data for ns/openshift-etcd...
2026-07-21T21:40:43.675750340Z Wrote inspect data to must-gather.
2026-07-21T21:40:43.675783691Z error: inspection completed with the errors occurred while gathering data:
2026-07-21T21:40:43.675783691Z     namespaces "assisted-installer" not found
2026-07-21T21:40:43.792025196Z Waiting on subprocesses to finish execution.
2026-07-21T21:40:43.809228563Z Executing Istio gather script
2026-07-21T21:40:43.817328781Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-21T21:40:43.824669986Z INFO: Collecting host service logs for crio
2026-07-21T21:40:43.824935176Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-21T21:40:43.825097902Z INFO: Collecting host service logs for kubelet
2026-07-21T21:40:43.825457324Z INFO: Collecting host service logs for rpm-ostreed
2026-07-21T21:40:43.825780349Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-21T21:40:43.825796440Z INFO: Waiting for node performance related collection to complete ...
2026-07-21T21:40:43.826154245Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-21T21:40:43.826498466Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-21T21:40:43.826831184Z INFO: Collecting host service logs for NetworkManager
2026-07-21T21:40:43.827175869Z INFO: Collecting host service logs for openvswitch
2026-07-21T21:40:43.827504199Z INFO: Collecting host service logs for ovs-configuration
2026-07-21T21:40:43.827838857Z INFO: Collecting host service logs for ovsdb-server
2026-07-21T21:40:43.828178913Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-21T21:40:43.828389781Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-21T21:40:43.828590862Z INFO: Waiting for worker host service log collection to complete ...
2026-07-21T21:40:43.830931514Z INFO: Gathering HAProxy config files
2026-07-21T21:40:44.077520052Z No resources found in openshift-etcd namespace.
2026-07-21T21:40:44.608978184Z error: the server doesn't have a resource type "performanceprofile"
2026-07-21T21:40:44.820566208Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-21T21:40:44.827396772Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-21T21:40:44.827432930Z INFO: on-disk MachineConfig config collection complete.
2026-07-21T21:40:44.832417892Z INFO: Collecting Insights Archives from insights-operator-555d9d7885-qgxmv
2026-07-21T21:40:44.832417892Z insights-runtime-extractor-8rrgw
2026-07-21T21:40:44.832417892Z insights-runtime-extractor-l4ddr
2026-07-21T21:40:44.832417892Z insights-runtime-extractor-sl5tb
2026-07-21T21:40:44.910404414Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-21T21:40:45.026272210Z ERROR: No running kube-apiserver pods found
2026-07-21T21:40:45.231310204Z INFO: "metallb-operator" not detected. Skipping.
2026-07-21T21:40:45.284314428Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-21T21:40:45.304579176Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-21T21:40:45.473548017Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-21T21:40:45.578274589Z No resources found
2026-07-21T21:40:45.638227257Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-21T21:40:45.641392889Z INFO: Worker host service log collection to complete.
2026-07-21T21:40:45.660100406Z Wrote inspect data to must-gather.
2026-07-21T21:40:45.786343680Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T21:40:45.890367084Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-21T21:40:45.958833567Z Wrote inspect data to must-gather.
2026-07-21T21:40:45.989680665Z No resources found
2026-07-21T21:40:46.051432449Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.066400835Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-21T21:40:46.066400835Z See 'oc rsync -h' for help and examples
2026-07-21T21:40:46.071310306Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.120243565Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.135114811Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.140282224Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.192351619Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T21:40:46.201375022Z INFO: 'previous-logs' folder not found on ip-10-0-135-153.ec2.internal, skipping...
2026-07-21T21:40:46.363559397Z error: the server doesn't have a resource type "machineconfigs"
2026-07-21T21:40:46.400403069Z tar: Removing leading `/' from member names
2026-07-21T21:40:46.411928933Z INFO: HAProxy config collection complete.
2026-07-21T21:40:46.418755580Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.424512533Z Gathering data for ns/openshift-monitoring...
2026-07-21T21:40:46.490476012Z INFO: Getting rules from prometheus-k8s-0
2026-07-21T21:40:46.547168798Z INFO: 'previous-logs' folder not found on ip-10-0-137-248.ec2.internal, skipping...
2026-07-21T21:40:46.578468114Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-21T21:40:46.698075406Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-21T21:40:46.723911021Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.821756073Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.834527409Z INFO: 'previous-logs' folder not found on ip-10-0-143-25.ec2.internal, skipping...
2026-07-21T21:40:46.834560212Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-21T21:40:46.834592873Z INFO: Machine Config Daemon termination log collection complete.
2026-07-21T21:40:46.969574192Z INFO: Getting status/config from prometheus-k8s-0
2026-07-21T21:40:46.979980363Z Wrote inspect data to must-gather.
2026-07-21T21:40:46.991814668Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-21T21:40:47.200913766Z Wrote inspect data to must-gather.
2026-07-21T21:40:47.274464825Z INFO: INTERCONNECT MODE
2026-07-21T21:40:47.274493855Z INFO: Gathering ovn-kubernetes DBs
2026-07-21T21:40:47.296278814Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-21T21:40:47.442404734Z Wrote inspect data to must-gather.
2026-07-21T21:40:47.465338442Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-21T21:40:47.477667870Z INFO: Gathering OVN_Northbound from ovnkube-node-28sbz...
2026-07-21T21:40:47.477931308Z INFO: Gathering OVN_Northbound from ovnkube-node-n5jxj...
2026-07-21T21:40:47.478204955Z INFO: Gathering OVN_Northbound from ovnkube-node-td9x9...
2026-07-21T21:40:47.478497935Z INFO: Gathering OVN_Southbound from ovnkube-node-28sbz...
2026-07-21T21:40:47.478759423Z INFO: Gathering OVN_Southbound from ovnkube-node-n5jxj...
2026-07-21T21:40:47.479017119Z INFO: Gathering OVN_Southbound from ovnkube-node-td9x9...
2026-07-21T21:40:47.509517646Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:47.780281310Z Wrote inspect data to must-gather.
2026-07-21T21:40:47.803642062Z tar: Removing leading `/' from member names
2026-07-21T21:40:47.812015552Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-21T21:40:47.828913721Z tar: Removing leading `/' from member names
2026-07-21T21:40:47.852677455Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:47.855890416Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-21T21:40:47.863686233Z tar: Removing leading `/' from member names
2026-07-21T21:40:47.875789621Z tar: Removing leading `/' from member names
2026-07-21T21:40:47.887055089Z tar: Removing leading `/' from member names
2026-07-21T21:40:47.895591951Z tar: Removing leading `/' from member names
2026-07-21T21:40:48.019901409Z Wrote inspect data to must-gather.
2026-07-21T21:40:48.085549980Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:48.192028167Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-21T21:40:48.244730026Z Wrote inspect data to must-gather.
2026-07-21T21:40:48.300859535Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:48.304178670Z INFO: Waiting for network log collection to complete ...
2026-07-21T21:40:48.304205843Z INFO: Waiting for ovnk database copies to complete ...
2026-07-21T21:40:48.304269084Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-21T21:40:48.304599516Z INFO: Copying ovnk databases complete.
2026-07-21T21:40:48.306279019Z 2.3M	must-gather/network_logs/ovnk_database_store
2026-07-21T21:40:48.308840108Z ovnk_database_store/
2026-07-21T21:40:48.308840108Z ovnk_database_store/ovnkube-node-n5jxj_nbdb
2026-07-21T21:40:48.315801347Z ovnk_database_store/ovnkube-node-28sbz_sbdb
2026-07-21T21:40:48.330435302Z ovnk_database_store/ovnkube-node-n5jxj_sbdb
2026-07-21T21:40:48.347125010Z ovnk_database_store/ovnkube-node-td9x9_sbdb
2026-07-21T21:40:48.361660943Z ovnk_database_store/ovnkube-node-td9x9_nbdb
2026-07-21T21:40:48.367037691Z ovnk_database_store/ovnkube-node-28sbz_nbdb
2026-07-21T21:40:48.374897034Z INFO: Network log collection complete.
2026-07-21T21:40:48.450011481Z Wrote inspect data to must-gather.
2026-07-21T21:40:48.492947959Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-21T21:40:48.500478208Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:48.670646469Z Wrote inspect data to must-gather.
2026-07-21T21:40:48.681907495Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-21T21:40:48.718303416Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:48.720507487Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-21T21:40:48.912298332Z INFO: Getting status from alertmanager-main-0
2026-07-21T21:40:48.940067143Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-21T21:40:48.943060246Z INFO: Fallback to identify the container image from release info
2026-07-21T21:40:48.947427947Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:49.150809379Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:49.153754556Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-21T21:40:49.276361268Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-21T21:40:49.307337568Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:49.440167576Z Gathering data for ns/openshift-network-console...
2026-07-21T21:40:49.498234688Z Wrote inspect data to /must-gather/istio.
2026-07-21T21:40:49.615371225Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bde4d3e6601b1a43249dcefbf210ae783590d4ff5957380d16c78b4d391ee9eb
2026-07-21T21:40:49.742579247Z daemonset.apps/perf-node-gather-daemonset created
2026-07-21T21:40:49.904423754Z Waiting for performance profile collector pods to become ready: 1
2026-07-21T21:40:49.993376144Z Gathering data for ns/openshift-console-operator...
2026-07-21T21:40:50.217262037Z Done executing Istio gather script
2026-07-21T21:40:50.439485075Z Gathering data for ns/openshift-console...
2026-07-21T21:40:50.983131885Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-21T21:40:51.022107153Z Waiting for performance profile collector pods to become ready: 2
2026-07-21T21:40:51.197675608Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T21:40:51.715701025Z Gathering data for ns/openshift-dns-operator...
2026-07-21T21:40:52.006320026Z Gathering data for ns/openshift-dns...
2026-07-21T21:40:52.129350733Z Waiting for performance profile collector pods to become ready: 3
2026-07-21T21:40:52.695723201Z Gathering data for ns/openshift-image-registry...
2026-07-21T21:40:53.248748289Z Waiting for performance profile collector pods to become ready: 4
2026-07-21T21:40:53.269079450Z Gathering data for ns/openshift-ingress-operator...
2026-07-21T21:40:53.633850174Z Gathering data for ns/openshift-ingress...
2026-07-21T21:40:53.987602259Z Gathering data for ns/openshift-ingress-canary...
2026-07-21T21:40:54.372432098Z Waiting for performance profile collector pods to become ready: 5
2026-07-21T21:40:54.433014853Z Gathering data for ns/openshift-insights...
2026-07-21T21:40:55.478739980Z Waiting for performance profile collector pods to become ready: 6
2026-07-21T21:40:56.204735003Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T21:40:56.379911074Z Gathering data for ns/openshift-config...
2026-07-21T21:40:56.602653294Z Waiting for performance profile collector pods to become ready: 7
2026-07-21T21:40:56.650026429Z Gathering data for ns/openshift-config-managed...
2026-07-21T21:40:57.145493279Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-21T21:40:57.412090014Z Gathering data for ns/openshift-kube-apiserver...
2026-07-21T21:40:57.702846415Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-21T21:40:57.733773847Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-21T21:40:57.985820471Z Collecting performance related data for node ip-10-0-137-248.ec2.internal
2026-07-21T21:40:57.993825086Z Collecting performance related data for node ip-10-0-143-25.ec2.internal
2026-07-21T21:40:58.001429603Z Collecting performance related data for node ip-10-0-135-153.ec2.internal
2026-07-21T21:40:58.018688725Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-21T21:40:58.422029883Z Gathering data for ns/openshift-kube-scheduler...
2026-07-21T21:40:58.917314530Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-21T21:40:59.355304332Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-21T21:40:59.818612323Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-21T21:41:00.175489386Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-21T21:41:00.736867936Z Gathering data for ns/openshift-multus...
2026-07-21T21:41:01.209380427Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T21:41:02.576632406Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-21T21:41:04.284992386Z Gathering data for ns/openshift-host-network...
2026-07-21T21:41:04.619191354Z Gathering data for ns/openshift-network-diagnostics...
2026-07-21T21:41:05.230195754Z Gathering data for ns/openshift-network-node-identity...
2026-07-21T21:41:05.763534071Z Gathering data for ns/openshift-network-operator...
2026-07-21T21:41:06.194687347Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-21T21:41:06.215426962Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T21:41:06.468861815Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-21T21:41:06.571617121Z Collecting kubelet logs for node ip-10-0-137-248.ec2.internal
2026-07-21T21:41:06.573357066Z Collecting kubelet logs for node ip-10-0-143-25.ec2.internal
2026-07-21T21:41:06.575068025Z Collecting kubelet logs for node ip-10-0-135-153.ec2.internal
2026-07-21T21:41:06.900156201Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-2q2l5 namespace
2026-07-21T21:41:06.907801097Z INFO: Node performance data collection complete.
2026-07-21T21:41:07.200261592Z Gathering data for ns/openshift-apiserver-operator...
2026-07-21T21:41:07.676809773Z Gathering data for ns/openshift-apiserver...
2026-07-21T21:41:08.120204310Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-21T21:41:08.428983197Z Gathering data for ns/openshift-controller-manager...
2026-07-21T21:41:08.832375853Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-21T21:41:09.675660902Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-21T21:41:10.145170962Z Gathering data for ns/openshift-service-ca-operator...
2026-07-21T21:41:10.620249718Z Gathering data for ns/openshift-service-ca...
2026-07-21T21:41:11.073976372Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-21T21:41:11.220111320Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-21T21:41:12.134902188Z Wrote inspect data to must-gather.
2026-07-21T21:41:12.134979162Z error: inspection completed with the errors occurred while gathering data:
2026-07-21T21:41:12.134979162Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-21T21:41:12.258138582Z error: the server doesn't have a resource type "clusters"
2026-07-21T21:41:12.378254915Z Caches written to disk