2026-07-09T18:28:54.274196630Z [disk usage checker] Started
2026-07-09T18:28:54.278570636Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:28:54.476593098Z Gathering data for ns/openshift-cluster-version...
2026-07-09T18:28:54.624375059Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:28:54.726607043Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:28:54.813730603Z Gathering data for ns/default...
2026-07-09T18:28:55.137234863Z Gathering data for ns/openshift...
2026-07-09T18:28:55.566972823Z Gathering data for ns/kube-system...
2026-07-09T18:28:56.380559689Z Gathering data for ns/openshift-etcd...
2026-07-09T18:28:56.713667212Z Wrote inspect data to must-gather.
2026-07-09T18:28:56.713700611Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:28:56.713700611Z     namespaces "assisted-installer" not found
2026-07-09T18:28:57.072573566Z Waiting on subprocesses to finish execution.
2026-07-09T18:28:57.076862143Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-09T18:28:57.085008368Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-09T18:28:57.086307308Z Executing Istio gather script
2026-07-09T18:28:57.090000902Z INFO: Gathering HAProxy config files
2026-07-09T18:28:57.093323909Z INFO: Collecting host service logs for crio
2026-07-09T18:28:57.093659569Z INFO: Collecting host service logs for kubelet
2026-07-09T18:28:57.093953763Z INFO: Collecting host service logs for rpm-ostreed
2026-07-09T18:28:57.094249523Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-09T18:28:57.094577625Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-09T18:28:57.094886722Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-09T18:28:57.095234758Z INFO: Collecting host service logs for NetworkManager
2026-07-09T18:28:57.095539362Z INFO: Collecting host service logs for openvswitch
2026-07-09T18:28:57.095665032Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-09T18:28:57.095875669Z INFO: Collecting host service logs for ovs-configuration
2026-07-09T18:28:57.096141086Z INFO: Collecting host service logs for ovsdb-server
2026-07-09T18:28:57.096482867Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-09T18:28:57.096871367Z INFO: Waiting for worker host service log collection to complete ...
2026-07-09T18:28:57.143750001Z INFO: Waiting for node performance related collection to complete ...
2026-07-09T18:28:57.510705927Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-09T18:28:57.510800433Z INFO: on-disk MachineConfig config collection complete.
2026-07-09T18:28:57.559715749Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-09T18:28:57.770270039Z No resources found
2026-07-09T18:28:57.802439758Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-09T18:28:57.982442391Z No resources found in openshift-etcd namespace.
2026-07-09T18:28:58.123052411Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-4tbk5
2026-07-09T18:28:58.123091406Z insights-runtime-extractor-4jwbn
2026-07-09T18:28:58.123109603Z insights-runtime-extractor-jjm7p
2026-07-09T18:28:58.123127907Z insights-runtime-extractor-kss8w
2026-07-09T18:28:58.168700264Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-09T18:28:58.286258103Z error: the server doesn't have a resource type "performanceprofile"
2026-07-09T18:28:58.517212261Z INFO: Worker host service log collection to complete.
2026-07-09T18:28:58.588060911Z INFO: "metallb-operator" not detected. Skipping.
2026-07-09T18:28:58.611477508Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-09T18:28:58.611477508Z See 'oc rsync -h' for help and examples
2026-07-09T18:28:58.659507327Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-09T18:28:58.713934921Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-09T18:28:58.721880960Z Wrote inspect data to must-gather.
2026-07-09T18:28:58.884720402Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-09T18:28:58.888850874Z Wrote inspect data to must-gather.
2026-07-09T18:28:58.944303187Z ERROR: No running kube-apiserver pods found
2026-07-09T18:28:59.174715279Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-09T18:28:59.304935104Z INFO: 'previous-logs' folder not found on ip-10-0-133-76.ec2.internal, skipping...
2026-07-09T18:28:59.333117778Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:28:59.345717279Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:28:59.355774782Z Gathering data for ns/openshift-monitoring...
2026-07-09T18:28:59.405893484Z Wrote inspect data to must-gather.
2026-07-09T18:28:59.406568413Z tar: Removing leading `/' from member names
2026-07-09T18:28:59.419335715Z INFO: HAProxy config collection complete.
2026-07-09T18:28:59.510943212Z Wrote inspect data to must-gather.
2026-07-09T18:28:59.513667512Z Wrote inspect data to must-gather.
2026-07-09T18:28:59.548267628Z No resources found
2026-07-09T18:28:59.696790938Z Wrote inspect data to must-gather.
2026-07-09T18:28:59.761676170Z Wrote inspect data to must-gather.
2026-07-09T18:28:59.781679477Z error: the server doesn't have a resource type "machineconfigs"
2026-07-09T18:28:59.823432675Z INFO: 'previous-logs' folder not found on ip-10-0-137-168.ec2.internal, skipping...
2026-07-09T18:28:59.846657831Z INFO: Getting rules from prometheus-k8s-0
2026-07-09T18:29:00.014004003Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-09T18:29:00.047335545Z Wrote inspect data to must-gather.
2026-07-09T18:29:00.141015928Z INFO: 'previous-logs' folder not found on ip-10-0-143-178.ec2.internal, skipping...
2026-07-09T18:29:00.141055067Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-09T18:29:00.141067112Z INFO: Machine Config Daemon termination log collection complete.
2026-07-09T18:29:00.286684038Z Wrote inspect data to must-gather.
2026-07-09T18:29:00.333619178Z Wrote inspect data to must-gather.
2026-07-09T18:29:00.337010145Z INFO: Getting status/config from prometheus-k8s-0
2026-07-09T18:29:00.457626212Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-09T18:29:00.623638954Z Wrote inspect data to must-gather.
2026-07-09T18:29:00.706022704Z Wrote inspect data to must-gather.
2026-07-09T18:29:00.771558372Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-09T18:29:00.832231619Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-09T18:29:00.946172749Z Wrote inspect data to must-gather.
2026-07-09T18:29:00.976635529Z INFO: INTERCONNECT MODE
2026-07-09T18:29:00.976759749Z INFO: Gathering ovn-kubernetes DBs
2026-07-09T18:29:01.116547307Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-09T18:29:01.169498131Z INFO: Gathering OVN_Northbound from ovnkube-node-6hhd7...
2026-07-09T18:29:01.169719637Z INFO: Gathering OVN_Northbound from ovnkube-node-p5dg8...
2026-07-09T18:29:01.169965009Z INFO: Gathering OVN_Northbound from ovnkube-node-sjxbv...
2026-07-09T18:29:01.170230260Z INFO: Gathering OVN_Southbound from ovnkube-node-6hhd7...
2026-07-09T18:29:01.170522456Z INFO: Gathering OVN_Southbound from ovnkube-node-p5dg8...
2026-07-09T18:29:01.170828016Z INFO: Gathering OVN_Southbound from ovnkube-node-sjxbv...
2026-07-09T18:29:01.218658401Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-09T18:29:01.235922099Z Wrote inspect data to must-gather.
2026-07-09T18:29:01.511486785Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-09T18:29:01.542073142Z tar: Removing leading `/' from member names
2026-07-09T18:29:01.544771970Z tar: Removing leading `/' from member names
2026-07-09T18:29:01.550470793Z tar: Removing leading `/' from member names
2026-07-09T18:29:01.576164959Z tar: Removing leading `/' from member names
2026-07-09T18:29:01.584226428Z tar: Removing leading `/' from member names
2026-07-09T18:29:01.609672807Z tar: Removing leading `/' from member names
2026-07-09T18:29:01.663069894Z Wrote inspect data to must-gather.
2026-07-09T18:29:01.779278994Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-09T18:29:01.798593578Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-09T18:29:01.915488682Z W0709 18:29:01.915441    1411 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-09T18:29:01.921985995Z Wrote inspect data to must-gather.
2026-07-09T18:29:02.023498670Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:02.041594313Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-09T18:29:02.166048719Z INFO: Waiting for network log collection to complete ...
2026-07-09T18:29:02.166090534Z INFO: Waiting for ovnk database copies to complete ...
2026-07-09T18:29:02.166612603Z INFO: Copying ovnk databases complete.
2026-07-09T18:29:02.168706943Z 3.5M	must-gather/network_logs/ovnk_database_store
2026-07-09T18:29:02.171069017Z ovnk_database_store/
2026-07-09T18:29:02.171069017Z ovnk_database_store/ovnkube-node-sjxbv_sbdb
2026-07-09T18:29:02.191956830Z ovnk_database_store/ovnkube-node-sjxbv_nbdb
2026-07-09T18:29:02.201003914Z ovnk_database_store/ovnkube-node-p5dg8_nbdb
2026-07-09T18:29:02.216703964Z ovnk_database_store/ovnkube-node-6hhd7_sbdb
2026-07-09T18:29:02.238847221Z ovnk_database_store/ovnkube-node-6hhd7_nbdb
2026-07-09T18:29:02.248643121Z ovnk_database_store/ovnkube-node-p5dg8_sbdb
2026-07-09T18:29:02.275526671Z INFO: Network log collection complete.
2026-07-09T18:29:02.278777196Z INFO: Getting status from alertmanager-main-0
2026-07-09T18:29:02.281596940Z Wrote inspect data to must-gather.
2026-07-09T18:29:02.282219563Z Gathering data for ns/openshift-network-console...
2026-07-09T18:29:02.292014462Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:02.295088176Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-09T18:29:02.512517922Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:02.575765958Z Wrote inspect data to must-gather.
2026-07-09T18:29:02.731280044Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:02.734968192Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-09T18:29:02.881901028Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-09T18:29:02.884469577Z INFO: Fallback to identify the container image from release info
2026-07-09T18:29:02.915652009Z Gathering data for ns/openshift-console-operator...
2026-07-09T18:29:02.951841441Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:03.157137104Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:03.159336347Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-09T18:29:03.383474828Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:03.436880220Z Gathering data for ns/openshift-console...
2026-07-09T18:29:03.469088897Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d
2026-07-09T18:29:03.605722478Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:03.608807785Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-09T18:29:03.626793500Z daemonset.apps/perf-node-gather-daemonset created
2026-07-09T18:29:03.762013959Z Waiting for performance profile collector pods to become ready: 1
2026-07-09T18:29:03.784534187Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:03.987749345Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:03.990969174Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-09T18:29:04.033424978Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-09T18:29:04.190648077Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:04.356453115Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:04.394565207Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:04.398108571Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-09T18:29:04.593645086Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:04.608100735Z Gathering data for ns/openshift-dns-operator...
2026-07-09T18:29:04.802289544Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:04.806269374Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-09T18:29:04.914906105Z Waiting for performance profile collector pods to become ready: 2
2026-07-09T18:29:04.963664475Z Gathering data for ns/openshift-dns...
2026-07-09T18:29:04.991704341Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:05.170842309Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:05.173345587Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-09T18:29:05.346640555Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:05.535241887Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:05.538567581Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-09T18:29:05.646418601Z Gathering data for ns/openshift-image-registry...
2026-07-09T18:29:05.711229903Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:05.904345554Z Wrote inspect data to /must-gather/istio.
2026-07-09T18:29:06.053065786Z Waiting for performance profile collector pods to become ready: 3
2026-07-09T18:29:06.182154496Z Gathering data for ns/openshift-ingress-operator...
2026-07-09T18:29:06.556434243Z Gathering data for ns/openshift-ingress...
2026-07-09T18:29:06.627284985Z Done executing Istio gather script
2026-07-09T18:29:06.918024291Z Gathering data for ns/openshift-ingress-canary...
2026-07-09T18:29:07.167161514Z Waiting for performance profile collector pods to become ready: 4
2026-07-09T18:29:07.351073694Z Gathering data for ns/openshift-insights...
2026-07-09T18:29:08.281389609Z Waiting for performance profile collector pods to become ready: 5
2026-07-09T18:29:09.361376403Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:09.405657840Z Waiting for performance profile collector pods to become ready: 6
2026-07-09T18:29:09.761972496Z Gathering data for ns/kserve...
2026-07-09T18:29:10.517068871Z Waiting for performance profile collector pods to become ready: 7
2026-07-09T18:29:11.264703737Z Gathering data for ns/openshift-config...
2026-07-09T18:29:11.599825072Z Gathering data for ns/openshift-config-managed...
2026-07-09T18:29:11.656008924Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-09T18:29:11.904441044Z Collecting performance related data for node ip-10-0-137-168.ec2.internal
2026-07-09T18:29:11.911436189Z Collecting performance related data for node ip-10-0-143-178.ec2.internal
2026-07-09T18:29:11.918649448Z Collecting performance related data for node ip-10-0-133-76.ec2.internal
2026-07-09T18:29:12.125160439Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-09T18:29:12.476590674Z Gathering data for ns/openshift-kube-apiserver...
2026-07-09T18:29:12.825020317Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-09T18:29:13.217696488Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-09T18:29:13.618833913Z Gathering data for ns/openshift-kube-scheduler...
2026-07-09T18:29:13.943514800Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-09T18:29:14.236838665Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-09T18:29:14.366517580Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:14.614782286Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-09T18:29:14.985002346Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-09T18:29:15.585827617Z Gathering data for ns/openshift-multus...
2026-07-09T18:29:17.097637098Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-09T18:29:18.565455482Z Gathering data for ns/openshift-host-network...
2026-07-09T18:29:18.932740918Z Gathering data for ns/openshift-network-diagnostics...
2026-07-09T18:29:19.371816275Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:19.573385268Z Gathering data for ns/openshift-network-node-identity...
2026-07-09T18:29:20.102167677Z Gathering data for ns/openshift-network-operator...
2026-07-09T18:29:20.356317581Z Collecting kubelet logs for node ip-10-0-137-168.ec2.internal
2026-07-09T18:29:20.358501297Z Collecting kubelet logs for node ip-10-0-143-178.ec2.internal
2026-07-09T18:29:20.360433357Z Collecting kubelet logs for node ip-10-0-133-76.ec2.internal
2026-07-09T18:29:20.559962693Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-09T18:29:20.773954003Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-5fmck namespace
2026-07-09T18:29:20.780684537Z INFO: Node performance data collection complete.
2026-07-09T18:29:20.892848544Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-09T18:29:21.453326081Z Gathering data for ns/openshift-apiserver-operator...
2026-07-09T18:29:21.791011819Z Gathering data for ns/openshift-apiserver...
2026-07-09T18:29:22.232713706Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-09T18:29:22.572030926Z Gathering data for ns/openshift-controller-manager...
2026-07-09T18:29:22.915536651Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-09T18:29:23.760356640Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-09T18:29:24.147919808Z Gathering data for ns/openshift-service-ca-operator...
2026-07-09T18:29:24.379906480Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70
2026-07-09T18:29:24.511912071Z Gathering data for ns/openshift-service-ca...
2026-07-09T18:29:24.915703021Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-09T18:29:25.876135906Z Wrote inspect data to must-gather.
2026-07-09T18:29:25.876187585Z error: inspection completed with the errors occurred while gathering data:
2026-07-09T18:29:25.876187585Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-09T18:29:26.016653655Z error: the server doesn't have a resource type "clusters"
2026-07-09T18:29:26.155950823Z Caches written to disk