2026-06-11T08:52:27.978646708Z [disk usage checker] Started
2026-06-11T08:52:27.982207778Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-06-11T08:52:28.165100278Z Gathering data for ns/openshift-cluster-version...
2026-06-11T08:52:28.281159306Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-11T08:52:28.377566394Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T08:52:28.462932547Z Gathering data for ns/default...
2026-06-11T08:52:28.746326014Z Gathering data for ns/openshift...
2026-06-11T08:52:29.175215169Z Gathering data for ns/kube-system...
2026-06-11T08:52:29.844974699Z Gathering data for ns/openshift-etcd...
2026-06-11T08:52:30.131657256Z Wrote inspect data to must-gather.
2026-06-11T08:52:30.131689706Z error: inspection completed with the errors occurred while gathering data:
2026-06-11T08:52:30.131689706Z     namespaces "assisted-installer" not found
2026-06-11T08:52:30.499747079Z Waiting on subprocesses to finish execution.
2026-06-11T08:52:30.502469776Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-11T08:52:30.509242307Z Executing Istio gather script
2026-06-11T08:52:30.509361844Z INFO: Gathering HAProxy config files
2026-06-11T08:52:30.510499633Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-11T08:52:30.518358530Z INFO: Collecting host service logs for crio
2026-06-11T08:52:30.518645677Z INFO: Collecting host service logs for kubelet
2026-06-11T08:52:30.518941788Z INFO: Collecting host service logs for rpm-ostreed
2026-06-11T08:52:30.519238389Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-11T08:52:30.519523534Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-11T08:52:30.519853548Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-11T08:52:30.520150555Z INFO: Collecting host service logs for NetworkManager
2026-06-11T08:52:30.520447792Z INFO: Collecting host service logs for openvswitch
2026-06-11T08:52:30.520749374Z INFO: Collecting host service logs for ovs-configuration
2026-06-11T08:52:30.521048795Z INFO: Collecting host service logs for ovsdb-server
2026-06-11T08:52:30.521405943Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-11T08:52:30.521969132Z INFO: Waiting for worker host service log collection to complete ...
2026-06-11T08:52:30.536067260Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-11T08:52:30.615122153Z INFO: Waiting for node performance related collection to complete ...
2026-06-11T08:52:31.037495335Z No resources found
2026-06-11T08:52:31.270843243Z error: the server doesn't have a resource type "performanceprofile"
2026-06-11T08:52:31.466272968Z Wrote inspect data to must-gather.
2026-06-11T08:52:31.508651750Z INFO: Collecting Insights Archives from insights-operator-896549976-lm5mp
2026-06-11T08:52:31.508804758Z insights-runtime-extractor-2vvtr
2026-06-11T08:52:31.508826259Z insights-runtime-extractor-9mkxx
2026-06-11T08:52:31.508846829Z insights-runtime-extractor-cm8nb
2026-06-11T08:52:31.533045770Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-11T08:52:31.533075997Z INFO: on-disk MachineConfig config collection complete.
2026-06-11T08:52:31.645248922Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-11T08:52:31.816455806Z INFO: "metallb-operator" not detected. Skipping.
2026-06-11T08:52:31.826529236Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-11T08:52:31.848234793Z No resources found in openshift-etcd namespace.
2026-06-11T08:52:31.961447014Z ERROR: No running kube-apiserver pods found
2026-06-11T08:52:32.113779289Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-11T08:52:32.288984332Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-11T08:52:32.340556706Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-11T08:52:32.370938247Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-11T08:52:32.370938247Z See 'oc rsync -h' for help and examples
2026-06-11T08:52:32.410980013Z INFO: Worker host service log collection to complete.
2026-06-11T08:52:32.512431734Z Wrote inspect data to must-gather.
2026-06-11T08:52:32.584049159Z Wrote inspect data to must-gather.
2026-06-11T08:52:32.609093800Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-11T08:52:32.707738524Z Wrote inspect data to must-gather.
2026-06-11T08:52:32.716081473Z INFO: 'previous-logs' folder not found on ip-10-0-133-143.ec2.internal, skipping...
2026-06-11T08:52:32.726766820Z Wrote inspect data to must-gather.
2026-06-11T08:52:32.738779885Z Wrote inspect data to must-gather.
2026-06-11T08:52:32.784464070Z No resources found
2026-06-11T08:52:32.802856707Z Wrote inspect data to must-gather.
2026-06-11T08:52:32.879948126Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T08:52:32.906298444Z Gathering data for ns/openshift-monitoring...
2026-06-11T08:52:32.992793139Z error: the server doesn't have a resource type "machineconfigs"
2026-06-11T08:52:32.997440956Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-11T08:52:33.007246217Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-06-11T08:52:33.062429896Z tar: Removing leading `/' from member names
2026-06-11T08:52:33.070100013Z INFO: HAProxy config collection complete.
2026-06-11T08:52:33.115330814Z INFO: 'previous-logs' folder not found on ip-10-0-140-201.ec2.internal, skipping...
2026-06-11T08:52:33.128390623Z Wrote inspect data to must-gather.
2026-06-11T08:52:33.197645396Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-11T08:52:33.242759982Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-11T08:52:33.394043824Z INFO: 'previous-logs' folder not found on ip-10-0-143-157.ec2.internal, skipping...
2026-06-11T08:52:33.394163851Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-11T08:52:33.394218050Z INFO: Machine Config Daemon termination log collection complete.
2026-06-11T08:52:33.398088424Z Wrote inspect data to must-gather.
2026-06-11T08:52:33.434993385Z Wrote inspect data to must-gather.
2026-06-11T08:52:33.444858107Z INFO: Getting rules from prometheus-k8s-0
2026-06-11T08:52:33.589053036Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-11T08:52:33.631127913Z Wrote inspect data to must-gather.
2026-06-11T08:52:33.797751187Z Wrote inspect data to must-gather.
2026-06-11T08:52:33.823613713Z INFO: Getting status/config from prometheus-k8s-0
2026-06-11T08:52:33.909907272Z INFO: INTERCONNECT MODE
2026-06-11T08:52:33.909907272Z INFO: Gathering ovn-kubernetes DBs
2026-06-11T08:52:33.989567021Z Wrote inspect data to must-gather.
2026-06-11T08:52:34.068565108Z INFO: Gathering OVN_Northbound from ovnkube-node-2w94q...
2026-06-11T08:52:34.068778006Z INFO: Gathering OVN_Northbound from ovnkube-node-rbjtp...
2026-06-11T08:52:34.069005583Z INFO: Gathering OVN_Northbound from ovnkube-node-z2zqf...
2026-06-11T08:52:34.069282091Z INFO: Gathering OVN_Southbound from ovnkube-node-2w94q...
2026-06-11T08:52:34.069561571Z INFO: Gathering OVN_Southbound from ovnkube-node-rbjtp...
2026-06-11T08:52:34.069880609Z INFO: Gathering OVN_Southbound from ovnkube-node-z2zqf...
2026-06-11T08:52:34.136754264Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-11T08:52:34.352284114Z Wrote inspect data to must-gather.
2026-06-11T08:52:34.354371127Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-11T08:52:34.366554066Z tar: Removing leading `/' from member names
2026-06-11T08:52:34.377851976Z tar: Removing leading `/' from member names
2026-06-11T08:52:34.383326083Z tar: Removing leading `/' from member names
2026-06-11T08:52:34.396577347Z tar: Removing leading `/' from member names
2026-06-11T08:52:34.432746290Z tar: Removing leading `/' from member names
2026-06-11T08:52:34.462920004Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:34.476173888Z tar: Removing leading `/' from member names
2026-06-11T08:52:34.613635974Z Wrote inspect data to must-gather.
2026-06-11T08:52:34.623323591Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-11T08:52:34.649029808Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-11T08:52:34.715869065Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:34.718806712Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-11T08:52:34.845797844Z Wrote inspect data to must-gather.
2026-06-11T08:52:34.877015573Z INFO: Waiting for network log collection to complete ...
2026-06-11T08:52:34.877060395Z INFO: Waiting for ovnk database copies to complete ...
2026-06-11T08:52:34.877573539Z INFO: Copying ovnk databases complete.
2026-06-11T08:52:34.877861964Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-11T08:52:34.879120812Z 2.5M	must-gather/network_logs/ovnk_database_store
2026-06-11T08:52:34.881496422Z ovnk_database_store/
2026-06-11T08:52:34.881496422Z ovnk_database_store/ovnkube-node-z2zqf_sbdb
2026-06-11T08:52:34.900988098Z ovnk_database_store/ovnkube-node-rbjtp_nbdb
2026-06-11T08:52:34.908825050Z ovnk_database_store/ovnkube-node-2w94q_sbdb
2026-06-11T08:52:34.927064213Z ovnk_database_store/ovnkube-node-z2zqf_nbdb
2026-06-11T08:52:34.934895921Z ovnk_database_store/ovnkube-node-2w94q_nbdb
2026-06-11T08:52:34.942504013Z ovnk_database_store/ovnkube-node-rbjtp_sbdb
2026-06-11T08:52:34.951188546Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:34.960172538Z INFO: Network log collection complete.
2026-06-11T08:52:35.073891298Z Wrote inspect data to must-gather.
2026-06-11T08:52:35.131405354Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-11T08:52:35.141043923Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:35.143107467Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-11T08:52:35.283656819Z W0611 08:52:35.283580    1521 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-11T08:52:35.288964176Z Wrote inspect data to must-gather.
2026-06-11T08:52:35.326499689Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-11T08:52:35.351994980Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:35.513734354Z Gathering data for ns/openshift-network-console...
2026-06-11T08:52:35.534637628Z INFO: Getting status from alertmanager-main-0
2026-06-11T08:52:35.576184042Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:35.579890591Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-11T08:52:35.584488771Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-11T08:52:35.586986836Z INFO: Fallback to identify the container image from release info
2026-06-11T08:52:35.803367031Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:35.983653755Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:35.984094360Z Gathering data for ns/openshift-console-operator...
2026-06-11T08:52:35.985880849Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-11T08:52:36.138699732Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:36.267407407Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-11T08:52:36.340513557Z Wrote inspect data to /must-gather/istio.
2026-06-11T08:52:36.383422819Z Gathering data for ns/openshift-console...
2026-06-11T08:52:36.405260004Z daemonset.apps/perf-node-gather-daemonset created
2026-06-11T08:52:36.545667713Z Waiting for performance profile collector pods to become ready: 1
2026-06-11T08:52:36.960179600Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-11T08:52:37.001208883Z Done executing Istio gather script
2026-06-11T08:52:37.427674945Z Gathering data for ns/openshift-dns-operator...
2026-06-11T08:52:37.662269937Z Waiting for performance profile collector pods to become ready: 2
2026-06-11T08:52:37.703342140Z Gathering data for ns/openshift-dns...
2026-06-11T08:52:38.014396165Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-06-11T08:52:38.286899886Z Gathering data for ns/openshift-image-registry...
2026-06-11T08:52:38.727997464Z Gathering data for ns/openshift-ingress-operator...
2026-06-11T08:52:38.775732685Z Waiting for performance profile collector pods to become ready: 3
2026-06-11T08:52:39.048675082Z Gathering data for ns/openshift-ingress...
2026-06-11T08:52:39.361282821Z Gathering data for ns/openshift-ingress-canary...
2026-06-11T08:52:39.757365869Z Gathering data for ns/openshift-insights...
2026-06-11T08:52:39.878480270Z Waiting for performance profile collector pods to become ready: 4
2026-06-11T08:52:40.986989478Z Waiting for performance profile collector pods to become ready: 5
2026-06-11T08:52:41.565874558Z Gathering data for ns/openshift-config...
2026-06-11T08:52:41.852350801Z Gathering data for ns/openshift-config-managed...
2026-06-11T08:52:42.085503365Z Waiting for performance profile collector pods to become ready: 6
2026-06-11T08:52:42.262437004Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-11T08:52:42.527993478Z Gathering data for ns/openshift-kube-apiserver...
2026-06-11T08:52:42.766803687Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-11T08:52:43.018773288Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-06-11T08:52:43.034383511Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-11T08:52:43.192518893Z Waiting for performance profile collector pods to become ready: 7
2026-06-11T08:52:43.311704746Z Gathering data for ns/openshift-kube-scheduler...
2026-06-11T08:52:43.548523709Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-11T08:52:43.862699108Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-11T08:52:44.287853612Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-11T08:52:44.307111201Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-11T08:52:44.547992356Z Collecting performance related data for node ip-10-0-140-201.ec2.internal
2026-06-11T08:52:44.556727535Z Collecting performance related data for node ip-10-0-143-157.ec2.internal
2026-06-11T08:52:44.563229265Z Collecting performance related data for node ip-10-0-133-143.ec2.internal
2026-06-11T08:52:44.667226139Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-11T08:52:45.244953907Z Gathering data for ns/openshift-multus...
2026-06-11T08:52:46.960164684Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-11T08:52:48.024698378Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-06-11T08:52:48.293471515Z Gathering data for ns/openshift-host-network...
2026-06-11T08:52:48.651996850Z Gathering data for ns/openshift-network-diagnostics...
2026-06-11T08:52:49.197758360Z Gathering data for ns/openshift-network-node-identity...
2026-06-11T08:52:49.597375029Z Gathering data for ns/openshift-network-operator...
2026-06-11T08:52:50.002511269Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-11T08:52:50.256037876Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-11T08:52:50.726245728Z Gathering data for ns/openshift-apiserver-operator...
2026-06-11T08:52:51.053381191Z Gathering data for ns/openshift-apiserver...
2026-06-11T08:52:51.449910285Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-11T08:52:51.745984679Z Gathering data for ns/openshift-controller-manager...
2026-06-11T08:52:52.038929609Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-11T08:52:52.469526451Z Collecting kubelet logs for node ip-10-0-140-201.ec2.internal
2026-06-11T08:52:52.471307873Z Collecting kubelet logs for node ip-10-0-143-157.ec2.internal
2026-06-11T08:52:52.472998190Z Collecting kubelet logs for node ip-10-0-133-143.ec2.internal
2026-06-11T08:52:52.736837848Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-11T08:52:52.813393134Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-tlrmq namespace
2026-06-11T08:52:52.818777930Z INFO: Node performance data collection complete.
2026-06-11T08:52:53.029707745Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70
2026-06-11T08:52:53.072799139Z Gathering data for ns/openshift-service-ca-operator...
2026-06-11T08:52:53.369376844Z Gathering data for ns/openshift-service-ca...
2026-06-11T08:52:53.708433928Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-11T08:52:54.580916517Z Wrote inspect data to must-gather.
2026-06-11T08:52:54.580958632Z error: inspection completed with the errors occurred while gathering data:
2026-06-11T08:52:54.580958632Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-11T08:52:54.695700301Z error: the server doesn't have a resource type "clusters"
2026-06-11T08:52:54.815038852Z Caches written to disk