2026-07-16T08:12:54.696875360Z [disk usage checker] Started
2026-07-16T08:12:54.700827777Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T08:12:54.896884192Z Gathering data for ns/openshift-cluster-version...
2026-07-16T08:12:55.030274106Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T08:12:55.121889673Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T08:12:55.205562974Z Gathering data for ns/default...
2026-07-16T08:12:55.508082173Z Gathering data for ns/openshift...
2026-07-16T08:12:55.925046315Z Gathering data for ns/kube-system...
2026-07-16T08:12:56.686053997Z Gathering data for ns/openshift-etcd...
2026-07-16T08:12:56.994603897Z Wrote inspect data to must-gather.
2026-07-16T08:12:56.994650856Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T08:12:56.994650856Z     namespaces "assisted-installer" not found
2026-07-16T08:12:57.260022254Z Waiting on subprocesses to finish execution.
2026-07-16T08:12:57.264539703Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-16T08:12:57.265841934Z INFO: Gathering HAProxy config files
2026-07-16T08:12:57.267509061Z Executing Istio gather script
2026-07-16T08:12:57.273780710Z INFO: Collecting host service logs for crio
2026-07-16T08:12:57.274057364Z INFO: Collecting host service logs for kubelet
2026-07-16T08:12:57.274341810Z INFO: Collecting host service logs for rpm-ostreed
2026-07-16T08:12:57.274685929Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-16T08:12:57.275115427Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-16T08:12:57.275457004Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-16T08:12:57.275807833Z INFO: Collecting host service logs for NetworkManager
2026-07-16T08:12:57.276105208Z INFO: Collecting host service logs for openvswitch
2026-07-16T08:12:57.276415245Z INFO: Collecting host service logs for ovs-configuration
2026-07-16T08:12:57.276740322Z INFO: Collecting host service logs for ovsdb-server
2026-07-16T08:12:57.277087782Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-16T08:12:57.277467168Z INFO: Waiting for worker host service log collection to complete ...
2026-07-16T08:12:57.304793695Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-16T08:12:57.355380208Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-16T08:12:57.370503546Z INFO: Waiting for node performance related collection to complete ...
2026-07-16T08:12:57.809676536Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-16T08:12:57.896154106Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-16T08:12:57.934834045Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-16T08:12:57.934876991Z INFO: on-disk MachineConfig config collection complete.
2026-07-16T08:12:57.978899754Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-16T08:12:58.116740451Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-sn9d2
2026-07-16T08:12:58.116905113Z insights-runtime-extractor-ttqw8
2026-07-16T08:12:58.116930531Z insights-runtime-extractor-vdz89
2026-07-16T08:12:58.116951503Z insights-runtime-extractor-xbgdl
2026-07-16T08:12:58.167094801Z No resources found
2026-07-16T08:12:58.287766538Z INFO: "metallb-operator" not detected. Skipping.
2026-07-16T08:12:58.335861325Z No resources found in openshift-etcd namespace.
2026-07-16T08:12:58.429442948Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-16T08:12:58.450881390Z ERROR: No running kube-apiserver pods found
2026-07-16T08:12:58.458888641Z Wrote inspect data to must-gather.
2026-07-16T08:12:58.522045402Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-16T08:12:58.634553577Z error: the server doesn't have a resource type "performanceprofile"
2026-07-16T08:12:58.649541326Z INFO: Worker host service log collection to complete.
2026-07-16T08:12:58.653965442Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-16T08:12:58.653965442Z See 'oc rsync -h' for help and examples
2026-07-16T08:12:58.662442425Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-16T08:12:59.317560885Z INFO: HAProxy config collection complete.
2026-07-16T08:12:59.326165001Z Wrote inspect data to must-gather.
2026-07-16T08:12:59.555132296Z Wrote inspect data to must-gather.
2026-07-16T08:12:59.623906124Z Wrote inspect data to must-gather.
2026-07-16T08:12:59.647756335Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-16T08:12:59.668217325Z Wrote inspect data to must-gather.
2026-07-16T08:12:59.687670160Z INFO: 'previous-logs' folder not found on ip-10-0-129-14.ec2.internal, skipping...
2026-07-16T08:12:59.720498959Z Wrote inspect data to must-gather.
2026-07-16T08:12:59.725075426Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T08:12:59.725231366Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T08:12:59.848780293Z INFO: Getting rules from prometheus-k8s-0
2026-07-16T08:12:59.889195107Z No resources found
2026-07-16T08:12:59.950821558Z error: the server doesn't have a resource type "machineconfigs"
2026-07-16T08:12:59.958364239Z Wrote inspect data to must-gather.
2026-07-16T08:12:59.962707591Z Gathering data for ns/openshift-monitoring...
2026-07-16T08:13:00.102746789Z INFO: 'previous-logs' folder not found on ip-10-0-133-134.ec2.internal, skipping...
2026-07-16T08:13:00.152168065Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-16T08:13:00.157048973Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-16T08:13:00.261402633Z Wrote inspect data to must-gather.
2026-07-16T08:13:00.318855188Z INFO: Getting status/config from prometheus-k8s-0
2026-07-16T08:13:00.390426912Z Wrote inspect data to must-gather.
2026-07-16T08:13:00.395437404Z INFO: 'previous-logs' folder not found on ip-10-0-138-63.ec2.internal, skipping...
2026-07-16T08:13:00.395465388Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-16T08:13:00.395478559Z INFO: Machine Config Daemon termination log collection complete.
2026-07-16T08:13:00.543260192Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-16T08:13:00.545137115Z Wrote inspect data to must-gather.
2026-07-16T08:13:00.753936580Z Wrote inspect data to must-gather.
2026-07-16T08:13:00.777834203Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-16T08:13:00.802652907Z Wrote inspect data to must-gather.
2026-07-16T08:13:00.958138694Z Wrote inspect data to must-gather.
2026-07-16T08:13:00.993919242Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-16T08:13:01.024161412Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-16T08:13:01.080614360Z INFO: INTERCONNECT MODE
2026-07-16T08:13:01.080678181Z INFO: Gathering ovn-kubernetes DBs
2026-07-16T08:13:01.163264786Z Wrote inspect data to must-gather.
2026-07-16T08:13:01.210571445Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:01.231355343Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-16T08:13:01.252272391Z INFO: Gathering OVN_Northbound from ovnkube-node-5fzj7...
2026-07-16T08:13:01.252521271Z INFO: Gathering OVN_Northbound from ovnkube-node-6p8bm...
2026-07-16T08:13:01.252780193Z INFO: Gathering OVN_Northbound from ovnkube-node-9fwvq...
2026-07-16T08:13:01.253093835Z INFO: Gathering OVN_Southbound from ovnkube-node-5fzj7...
2026-07-16T08:13:01.253407501Z INFO: Gathering OVN_Southbound from ovnkube-node-6p8bm...
2026-07-16T08:13:01.253739776Z INFO: Gathering OVN_Southbound from ovnkube-node-9fwvq...
2026-07-16T08:13:01.485820464Z Wrote inspect data to must-gather.
2026-07-16T08:13:01.557111432Z tar: Removing leading `/' from member names
2026-07-16T08:13:01.573366775Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-16T08:13:01.588272111Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:01.595132012Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-16T08:13:01.639031306Z tar: Removing leading `/' from member names
2026-07-16T08:13:01.639398882Z tar: Removing leading `/' from member names
2026-07-16T08:13:01.650241187Z tar: Removing leading `/' from member names
2026-07-16T08:13:01.652794959Z tar: Removing leading `/' from member names
2026-07-16T08:13:01.657759523Z tar: Removing leading `/' from member names
2026-07-16T08:13:01.702216363Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-16T08:13:01.800960861Z Wrote inspect data to must-gather.
2026-07-16T08:13:01.831241725Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:02.024178621Z INFO: Getting status from alertmanager-main-0
2026-07-16T08:13:02.059884322Z Wrote inspect data to must-gather.
2026-07-16T08:13:02.106958703Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:02.109810736Z INFO: Waiting for network log collection to complete ...
2026-07-16T08:13:02.109810736Z INFO: Waiting for ovnk database copies to complete ...
2026-07-16T08:13:02.110326318Z INFO: Copying ovnk databases complete.
2026-07-16T08:13:02.110598969Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-16T08:13:02.113110115Z 2.9M	must-gather/network_logs/ovnk_database_store
2026-07-16T08:13:02.115517747Z ovnk_database_store/
2026-07-16T08:13:02.115517747Z ovnk_database_store/ovnkube-node-9fwvq_nbdb
2026-07-16T08:13:02.122507845Z ovnk_database_store/ovnkube-node-6p8bm_nbdb
2026-07-16T08:13:02.131652329Z ovnk_database_store/ovnkube-node-6p8bm_sbdb
2026-07-16T08:13:02.150091155Z ovnk_database_store/ovnkube-node-5fzj7_nbdb
2026-07-16T08:13:02.158685111Z ovnk_database_store/ovnkube-node-5fzj7_sbdb
2026-07-16T08:13:02.178782520Z ovnk_database_store/ovnkube-node-9fwvq_sbdb
2026-07-16T08:13:02.200116287Z INFO: Network log collection complete.
2026-07-16T08:13:02.276564015Z W0716 08:13:02.276523    1541 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-16T08:13:02.282016354Z Wrote inspect data to must-gather.
2026-07-16T08:13:02.321129547Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:02.376327996Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-16T08:13:02.521616252Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:02.524142701Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-16T08:13:02.534262021Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-16T08:13:02.536752051Z INFO: Fallback to identify the container image from release info
2026-07-16T08:13:02.559183112Z Gathering data for ns/openshift-network-console...
2026-07-16T08:13:02.735721811Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:02.927117531Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:02.930411776Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-16T08:13:02.990845326Z Gathering data for ns/openshift-console-operator...
2026-07-16T08:13:03.100835853Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:03.185387189Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-16T08:13:03.304209425Z Wrote inspect data to /must-gather/istio.
2026-07-16T08:13:03.321843984Z daemonset.apps/perf-node-gather-daemonset created
2026-07-16T08:13:03.423087879Z Gathering data for ns/openshift-console...
2026-07-16T08:13:03.453255204Z Waiting for performance profile collector pods to become ready: 1
2026-07-16T08:13:03.987068624Z Done executing Istio gather script
2026-07-16T08:13:03.999611247Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-16T08:13:04.497988285Z Gathering data for ns/openshift-dns-operator...
2026-07-16T08:13:04.584651293Z Waiting for performance profile collector pods to become ready: 2
2026-07-16T08:13:04.745393404Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T08:13:04.791534131Z Gathering data for ns/openshift-dns...
2026-07-16T08:13:05.388042260Z Gathering data for ns/openshift-image-registry...
2026-07-16T08:13:05.683142369Z Waiting for performance profile collector pods to become ready: 3
2026-07-16T08:13:05.870579768Z Gathering data for ns/openshift-ingress-operator...
2026-07-16T08:13:06.261366012Z Gathering data for ns/openshift-ingress...
2026-07-16T08:13:06.613808426Z Gathering data for ns/openshift-ingress-canary...
2026-07-16T08:13:06.803224683Z Waiting for performance profile collector pods to become ready: 4
2026-07-16T08:13:07.040342295Z Gathering data for ns/openshift-insights...
2026-07-16T08:13:07.908196232Z Waiting for performance profile collector pods to become ready: 5
2026-07-16T08:13:08.950281451Z Gathering data for ns/openshift-jobset-operator...
2026-07-16T08:13:09.009336861Z Waiting for performance profile collector pods to become ready: 6
2026-07-16T08:13:09.751891945Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T08:13:09.755356572Z Gathering data for ns/openshift-config...
2026-07-16T08:13:10.025846387Z Gathering data for ns/openshift-config-managed...
2026-07-16T08:13:10.126656928Z Waiting for performance profile collector pods to become ready: 7
2026-07-16T08:13:10.462388032Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-16T08:13:10.729210393Z Gathering data for ns/openshift-kube-apiserver...
2026-07-16T08:13:11.032327559Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-16T08:13:11.237039447Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-16T08:13:11.352498448Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-16T08:13:11.488643104Z Collecting performance related data for node ip-10-0-129-14.ec2.internal
2026-07-16T08:13:11.495224881Z Collecting performance related data for node ip-10-0-138-63.ec2.internal
2026-07-16T08:13:11.502278199Z Collecting performance related data for node ip-10-0-133-134.ec2.internal
2026-07-16T08:13:11.709074708Z Gathering data for ns/openshift-kube-scheduler...
2026-07-16T08:13:12.022506087Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-16T08:13:12.413363075Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-16T08:13:12.838555866Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-16T08:13:13.251649898Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-16T08:13:13.863920989Z Gathering data for ns/openshift-multus...
2026-07-16T08:13:14.757232372Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T08:13:15.407793738Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-16T08:13:16.743898786Z Gathering data for ns/openshift-host-network...
2026-07-16T08:13:17.071001463Z Gathering data for ns/openshift-network-diagnostics...
2026-07-16T08:13:17.689657300Z Gathering data for ns/openshift-network-node-identity...
2026-07-16T08:13:18.124396762Z Gathering data for ns/openshift-network-operator...
2026-07-16T08:13:18.533295050Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-16T08:13:18.926837270Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-16T08:13:19.441899508Z Gathering data for ns/openshift-apiserver-operator...
2026-07-16T08:13:19.527422862Z Collecting kubelet logs for node ip-10-0-129-14.ec2.internal
2026-07-16T08:13:19.529319972Z Collecting kubelet logs for node ip-10-0-138-63.ec2.internal
2026-07-16T08:13:19.530979250Z Collecting kubelet logs for node ip-10-0-133-134.ec2.internal
2026-07-16T08:13:19.763153261Z [disk usage checker] Volume usage percentage: current = 17 ; allowed = 70
2026-07-16T08:13:19.845981715Z Gathering data for ns/openshift-apiserver...
2026-07-16T08:13:19.848074302Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-sfb7j namespace
2026-07-16T08:13:19.852862360Z INFO: Node performance data collection complete.
2026-07-16T08:13:20.327063918Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-16T08:13:20.593379436Z Gathering data for ns/openshift-controller-manager...
2026-07-16T08:13:20.875911877Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-16T08:13:21.500874944Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-16T08:13:21.790903298Z Gathering data for ns/openshift-service-ca-operator...
2026-07-16T08:13:22.101274870Z Gathering data for ns/openshift-service-ca...
2026-07-16T08:13:22.446073126Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-16T08:13:23.337731701Z Wrote inspect data to must-gather.
2026-07-16T08:13:23.337771524Z error: inspection completed with the errors occurred while gathering data:
2026-07-16T08:13:23.337771524Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-16T08:13:23.462608378Z error: the server doesn't have a resource type "clusters"
2026-07-16T08:13:23.600003307Z Caches written to disk