2026-06-04T23:58:35.459871830Z [disk usage checker] Started
2026-06-04T23:58:35.463792843Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-04T23:58:35.704702028Z Gathering data for ns/openshift-cluster-version...
2026-06-04T23:58:35.893268020Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-04T23:58:36.014561577Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-04T23:58:36.117482794Z Gathering data for ns/default...
2026-06-04T23:58:36.487777157Z Gathering data for ns/openshift...
2026-06-04T23:58:36.945834456Z Gathering data for ns/kube-system...
2026-06-04T23:58:37.781671790Z Gathering data for ns/openshift-etcd...
2026-06-04T23:58:38.113186700Z Wrote inspect data to must-gather.
2026-06-04T23:58:38.113221632Z error: inspection completed with the errors occurred while gathering data:
2026-06-04T23:58:38.113221632Z     namespaces "assisted-installer" not found
2026-06-04T23:58:38.227588655Z Waiting on subprocesses to finish execution.
2026-06-04T23:58:38.235562888Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-04T23:58:38.238110894Z INFO: Gathering HAProxy config files
2026-06-04T23:58:38.238817464Z INFO: Collecting host service logs for crio
2026-06-04T23:58:38.239138414Z INFO: Collecting host service logs for kubelet
2026-06-04T23:58:38.239467208Z INFO: Collecting host service logs for rpm-ostreed
2026-06-04T23:58:38.239781150Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-04T23:58:38.240135858Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-04T23:58:38.240441036Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-04T23:58:38.240741991Z INFO: Collecting host service logs for NetworkManager
2026-06-04T23:58:38.241186222Z INFO: Collecting host service logs for openvswitch
2026-06-04T23:58:38.241533759Z INFO: Collecting host service logs for ovs-configuration
2026-06-04T23:58:38.241904065Z INFO: Collecting host service logs for ovsdb-server
2026-06-04T23:58:38.242257326Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-04T23:58:38.242679306Z INFO: Waiting for worker host service log collection to complete ...
2026-06-04T23:58:38.246297651Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-04T23:58:38.246860792Z Executing Istio gather script
2026-06-04T23:58:38.254489102Z INFO: Waiting for node performance related collection to complete ...
2026-06-04T23:58:38.272718220Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-04T23:58:38.675937014Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-04T23:58:38.907519989Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-04T23:58:38.907615387Z INFO: on-disk MachineConfig config collection complete.
2026-06-04T23:58:39.080642722Z No resources found
2026-06-04T23:58:39.199214671Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-04T23:58:39.228340349Z INFO: "metallb-operator" not detected. Skipping.
2026-06-04T23:58:39.339177161Z error: the server doesn't have a resource type "performanceprofile"
2026-06-04T23:58:39.370375882Z INFO: Collecting Insights Archives from insights-operator-896549976-4726g
2026-06-04T23:58:39.370375882Z insights-runtime-extractor-m5cnc
2026-06-04T23:58:39.370375882Z insights-runtime-extractor-s2wmx
2026-06-04T23:58:39.370375882Z insights-runtime-extractor-sjdzd
2026-06-04T23:58:39.858009237Z No resources found in openshift-etcd namespace.
2026-06-04T23:58:39.962659539Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.153870489Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-04T23:58:40.230673900Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.270481734Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-04T23:58:40.274015019Z No resources found
2026-06-04T23:58:40.276017719Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.277629750Z INFO: Worker host service log collection to complete.
2026-06-04T23:58:40.307006609Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-04T23:58:40.344523871Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-04T23:58:40.344523871Z See 'oc rsync -h' for help and examples
2026-06-04T23:58:40.447352726Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-04T23:58:40.486450573Z Gathering data for ns/openshift-monitoring...
2026-06-04T23:58:40.545145843Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-04T23:58:40.553081866Z ERROR: No running kube-apiserver pods found
2026-06-04T23:58:40.594664372Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.643385573Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.707692114Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-04T23:58:40.768036388Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.793836735Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.803749014Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-04T23:58:40.816928762Z Wrote inspect data to must-gather.
2026-06-04T23:58:40.835533586Z INFO: 'previous-logs' folder not found on ip-10-0-129-155.ec2.internal, skipping...
2026-06-04T23:58:41.000739100Z tar: Removing leading `/' from member names
2026-06-04T23:58:41.007217164Z INFO: HAProxy config collection complete.
2026-06-04T23:58:41.020187427Z error: the server doesn't have a resource type "machineconfigs"
2026-06-04T23:58:41.026087469Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-04T23:58:41.122257157Z Wrote inspect data to must-gather.
2026-06-04T23:58:41.128499853Z INFO: 'previous-logs' folder not found on ip-10-0-131-209.ec2.internal, skipping...
2026-06-04T23:58:41.186388517Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-04T23:58:41.246244723Z INFO: Getting rules from prometheus-k8s-0
2026-06-04T23:58:41.406603309Z Wrote inspect data to must-gather.
2026-06-04T23:58:41.410559668Z INFO: 'previous-logs' folder not found on ip-10-0-135-207.ec2.internal, skipping...
2026-06-04T23:58:41.410591261Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-04T23:58:41.410624642Z INFO: Machine Config Daemon termination log collection complete.
2026-06-04T23:58:41.451065689Z Wrote inspect data to must-gather.
2026-06-04T23:58:41.614653501Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-04T23:58:41.662933942Z INFO: Getting status/config from prometheus-k8s-0
2026-06-04T23:58:41.680652672Z INFO: INTERCONNECT MODE
2026-06-04T23:58:41.680683966Z INFO: Gathering ovn-kubernetes DBs
2026-06-04T23:58:41.742479433Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-04T23:58:41.859415338Z INFO: Gathering OVN_Northbound from ovnkube-node-jj9b2...
2026-06-04T23:58:41.859772920Z INFO: Gathering OVN_Northbound from ovnkube-node-qmlgm...
2026-06-04T23:58:41.860084709Z INFO: Gathering OVN_Northbound from ovnkube-node-sk785...
2026-06-04T23:58:41.860598838Z INFO: Gathering OVN_Southbound from ovnkube-node-jj9b2...
2026-06-04T23:58:41.860905112Z INFO: Gathering OVN_Southbound from ovnkube-node-qmlgm...
2026-06-04T23:58:41.860927701Z Wrote inspect data to must-gather.
2026-06-04T23:58:41.861216939Z INFO: Gathering OVN_Southbound from ovnkube-node-sk785...
2026-06-04T23:58:41.999244456Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:42.020077356Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-04T23:58:42.195058218Z tar: Removing leading `/' from member names
2026-06-04T23:58:42.210642164Z tar: Removing leading `/' from member names
2026-06-04T23:58:42.211679405Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-04T23:58:42.234794175Z tar: Removing leading `/' from member names
2026-06-04T23:58:42.251594210Z tar: Removing leading `/' from member names
2026-06-04T23:58:42.252449331Z tar: Removing leading `/' from member names
2026-06-04T23:58:42.258796195Z Wrote inspect data to must-gather.
2026-06-04T23:58:42.275740952Z tar: Removing leading `/' from member names
2026-06-04T23:58:42.333526796Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:42.336461012Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-04T23:58:42.468117605Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-04T23:58:42.507648787Z Wrote inspect data to must-gather.
2026-06-04T23:58:42.575336140Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:42.710191444Z INFO: Waiting for network log collection to complete ...
2026-06-04T23:58:42.710251521Z INFO: Waiting for ovnk database copies to complete ...
2026-06-04T23:58:42.710790543Z INFO: Copying ovnk databases complete.
2026-06-04T23:58:42.713163869Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-06-04T23:58:42.716160025Z ovnk_database_store/
2026-06-04T23:58:42.716160025Z ovnk_database_store/ovnkube-node-sk785_sbdb
2026-06-04T23:58:42.727005474Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-04T23:58:42.740951956Z ovnk_database_store/ovnkube-node-qmlgm_sbdb
2026-06-04T23:58:42.755207459Z W0604 23:58:42.755160    1427 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-04T23:58:42.756922780Z ovnk_database_store/ovnkube-node-sk785_nbdb
2026-06-04T23:58:42.762635977Z Wrote inspect data to must-gather.
2026-06-04T23:58:42.763021766Z ovnk_database_store/ovnkube-node-jj9b2_nbdb
2026-06-04T23:58:42.771797451Z ovnk_database_store/ovnkube-node-qmlgm_nbdb
2026-06-04T23:58:42.773230347Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-04T23:58:42.778670009Z ovnk_database_store/ovnkube-node-jj9b2_sbdb
2026-06-04T23:58:42.796757689Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:42.799334022Z INFO: Network log collection complete.
2026-06-04T23:58:42.800743718Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-04T23:58:42.966046974Z Gathering data for ns/openshift-network-console...
2026-06-04T23:58:42.999265180Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:43.000755240Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-04T23:58:43.087460522Z Wrote inspect data to must-gather.
2026-06-04T23:58:43.230263040Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:43.232883073Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-04T23:58:43.415750323Z Wrote inspect data to must-gather.
2026-06-04T23:58:43.417540740Z INFO: Getting status from alertmanager-main-0
2026-06-04T23:58:43.446582440Z Gathering data for ns/openshift-console-operator...
2026-06-04T23:58:43.485147813Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:43.711908858Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:43.715687761Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-04T23:58:43.742509329Z Wrote inspect data to must-gather.
2026-06-04T23:58:43.866086370Z Gathering data for ns/openshift-console...
2026-06-04T23:58:43.904404562Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:44.016560115Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-04T23:58:44.021144559Z INFO: Fallback to identify the container image from release info
2026-06-04T23:58:44.099062568Z Wrote inspect data to /must-gather/istio.
2026-06-04T23:58:44.376225002Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-04T23:58:44.762659343Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-04T23:58:44.772527031Z Done executing Istio gather script
2026-06-04T23:58:44.859271858Z Gathering data for ns/openshift-dns-operator...
2026-06-04T23:58:44.889686546Z daemonset.apps/perf-node-gather-daemonset created
2026-06-04T23:58:45.015491825Z Waiting for performance profile collector pods to become ready: 1
2026-06-04T23:58:45.128303934Z Gathering data for ns/openshift-dns...
2026-06-04T23:58:45.553957968Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-04T23:58:45.681109052Z Gathering data for ns/openshift-image-registry...
2026-06-04T23:58:46.109134263Z Gathering data for ns/openshift-ingress-operator...
2026-06-04T23:58:46.145883404Z Waiting for performance profile collector pods to become ready: 2
2026-06-04T23:58:46.497235721Z Gathering data for ns/openshift-ingress...
2026-06-04T23:58:46.815314023Z Gathering data for ns/openshift-ingress-canary...
2026-06-04T23:58:47.156037836Z Gathering data for ns/openshift-insights...
2026-06-04T23:58:47.258406891Z Waiting for performance profile collector pods to become ready: 3
2026-06-04T23:58:48.373202136Z Waiting for performance profile collector pods to become ready: 4
2026-06-04T23:58:48.898954859Z Gathering data for ns/openshift-config...
2026-06-04T23:58:49.131470067Z Gathering data for ns/openshift-config-managed...
2026-06-04T23:58:49.488464709Z Waiting for performance profile collector pods to become ready: 5
2026-06-04T23:58:49.517896038Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-04T23:58:49.796288075Z Gathering data for ns/openshift-kube-apiserver...
2026-06-04T23:58:50.030094638Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-04T23:58:50.283118586Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-04T23:58:50.534359841Z Gathering data for ns/openshift-kube-scheduler...
2026-06-04T23:58:50.562046768Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-04T23:58:50.611837261Z Waiting for performance profile collector pods to become ready: 6
2026-06-04T23:58:50.778622452Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-04T23:58:51.036386902Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-04T23:58:51.323971477Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-04T23:58:51.599831876Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-04T23:58:51.731510968Z Waiting for performance profile collector pods to become ready: 7
2026-06-04T23:58:52.051082879Z Gathering data for ns/openshift-multus...
2026-06-04T23:58:52.839607812Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-04T23:58:53.106191355Z Collecting performance related data for node ip-10-0-131-209.ec2.internal
2026-06-04T23:58:53.115103738Z Collecting performance related data for node ip-10-0-135-207.ec2.internal
2026-06-04T23:58:53.125930757Z Collecting performance related data for node ip-10-0-129-155.ec2.internal
2026-06-04T23:58:53.375552685Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-04T23:58:54.668195687Z Gathering data for ns/openshift-host-network...
2026-06-04T23:58:54.980971186Z Gathering data for ns/openshift-network-diagnostics...
2026-06-04T23:58:55.480308238Z Gathering data for ns/openshift-network-node-identity...
2026-06-04T23:58:55.567236298Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-04T23:58:55.927176057Z Gathering data for ns/openshift-network-operator...
2026-06-04T23:58:56.320310744Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-04T23:58:56.566584681Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-04T23:58:57.024980784Z Gathering data for ns/openshift-apiserver-operator...
2026-06-04T23:58:57.319115738Z Gathering data for ns/openshift-apiserver...
2026-06-04T23:58:57.702841590Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-04T23:58:57.964258736Z Gathering data for ns/openshift-controller-manager...
2026-06-04T23:58:58.215107995Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-04T23:58:58.857251982Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-04T23:58:59.161064259Z Gathering data for ns/openshift-service-ca-operator...
2026-06-04T23:58:59.452461361Z Gathering data for ns/openshift-service-ca...
2026-06-04T23:58:59.787706014Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-04T23:59:00.572921959Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-04T23:59:00.631795324Z Wrote inspect data to must-gather.
2026-06-04T23:59:00.631839269Z error: inspection completed with the errors occurred while gathering data:
2026-06-04T23:59:00.631839269Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-04T23:59:01.521386841Z Collecting kubelet logs for node ip-10-0-131-209.ec2.internal
2026-06-04T23:59:01.523256866Z Collecting kubelet logs for node ip-10-0-135-207.ec2.internal
2026-06-04T23:59:01.524956521Z Collecting kubelet logs for node ip-10-0-129-155.ec2.internal
2026-06-04T23:59:01.822910309Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-42lxq namespace
2026-06-04T23:59:01.827960930Z INFO: Node performance data collection complete.
2026-06-04T23:59:01.955201824Z error: the server doesn't have a resource type "clusters"
2026-06-04T23:59:02.110872339Z Caches written to disk