2026-06-08T19:31:31.574017167Z [disk usage checker] Started
2026-06-08T19:31:31.577831154Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T19:31:31.768827211Z Gathering data for ns/openshift-cluster-version...
2026-06-08T19:31:31.911019676Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-08T19:31:31.996237561Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-08T19:31:32.076279796Z Gathering data for ns/default...
2026-06-08T19:31:32.353930480Z Gathering data for ns/openshift...
2026-06-08T19:31:32.737675296Z Gathering data for ns/kube-system...
2026-06-08T19:31:33.420012272Z Gathering data for ns/openshift-etcd...
2026-06-08T19:31:33.706948448Z Wrote inspect data to must-gather.
2026-06-08T19:31:33.706978128Z error: inspection completed with the errors occurred while gathering data:
2026-06-08T19:31:33.706978128Z     namespaces "assisted-installer" not found
2026-06-08T19:31:34.065727984Z Waiting on subprocesses to finish execution.
2026-06-08T19:31:34.074505866Z INFO: Gathering machine config daemon's old logs from all nodes
2026-06-08T19:31:34.078643118Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-06-08T19:31:34.082837462Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-06-08T19:31:34.087468678Z INFO: Gathering HAProxy config files
2026-06-08T19:31:34.091871305Z Executing Istio gather script
2026-06-08T19:31:34.121564919Z INFO: Collecting host service logs for crio
2026-06-08T19:31:34.128078094Z INFO: Collecting host service logs for kubelet
2026-06-08T19:31:34.128676341Z INFO: Collecting host service logs for rpm-ostreed
2026-06-08T19:31:34.129047365Z INFO: Collecting host service logs for ostree-finalize-staged
2026-06-08T19:31:34.132745290Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-06-08T19:31:34.133174920Z INFO: Collecting host service logs for machine-config-daemon-host
2026-06-08T19:31:34.133558930Z INFO: Collecting host service logs for NetworkManager
2026-06-08T19:31:34.133934147Z INFO: Collecting host service logs for openvswitch
2026-06-08T19:31:34.134287221Z INFO: Collecting host service logs for ovs-configuration
2026-06-08T19:31:34.134663603Z INFO: Collecting host service logs for ovsdb-server
2026-06-08T19:31:34.135021670Z INFO: Collecting host service logs for ovs-vswitchd
2026-06-08T19:31:34.135447824Z INFO: Waiting for worker host service log collection to complete ...
2026-06-08T19:31:34.185123001Z INFO: Waiting for node performance related collection to complete ...
2026-06-08T19:31:34.812382020Z INFO: Collecting Insights Archives from insights-operator-896549976-4rqfw
2026-06-08T19:31:34.812437548Z insights-runtime-extractor-cgdtt
2026-06-08T19:31:34.812456410Z insights-runtime-extractor-f645z
2026-06-08T19:31:34.812474909Z insights-runtime-extractor-vljjm
2026-06-08T19:31:35.052467551Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-06-08T19:31:35.052560884Z INFO: on-disk MachineConfig config collection complete.
2026-06-08T19:31:35.121656209Z INFO: Found 1 replicas - prometheus-k8s-0
2026-06-08T19:31:35.122865267Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-06-08T19:31:35.167066034Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-06-08T19:31:35.178966325Z No resources found
2026-06-08T19:31:35.219716656Z INFO: "metallb-operator" not detected. Skipping.
2026-06-08T19:31:35.294042838Z No resources found in openshift-etcd namespace.
2026-06-08T19:31:35.306774127Z INFO: "sriov-network-operator" not detected. Skipping.
2026-06-08T19:31:35.382141515Z Wrote inspect data to must-gather.
2026-06-08T19:31:35.586506949Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-06-08T19:31:35.586506949Z See 'oc rsync -h' for help and examples
2026-06-08T19:31:35.637856864Z error: the server doesn't have a resource type "performanceprofile"
2026-06-08T19:31:35.859499776Z Wrote inspect data to must-gather.
2026-06-08T19:31:35.876566485Z INFO: Worker host service log collection to complete.
2026-06-08T19:31:36.017472404Z ERROR: No running kube-apiserver pods found
2026-06-08T19:31:36.039697426Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-06-08T19:31:36.120743108Z Gathering data for ns/openshift-monitoring...
2026-06-08T19:31:36.284029888Z No resources found
2026-06-08T19:31:36.361440109Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-06-08T19:31:36.443295858Z Wrote inspect data to must-gather.
2026-06-08T19:31:36.474599915Z INFO: 'previous-logs' folder not found on ip-10-0-128-82.ec2.internal, skipping...
2026-06-08T19:31:36.486667022Z Wrote inspect data to must-gather.
2026-06-08T19:31:36.502068195Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-08T19:31:36.519161207Z Wrote inspect data to must-gather.
2026-06-08T19:31:36.520576411Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-06-08T19:31:36.544744563Z INFO: Waiting for HAProxy config collection to complete ...
2026-06-08T19:31:36.551508468Z Wrote inspect data to must-gather.
2026-06-08T19:31:36.592288685Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T19:31:36.668773685Z Wrote inspect data to must-gather.
2026-06-08T19:31:36.776826986Z Wrote inspect data to must-gather.
2026-06-08T19:31:36.780709538Z tar: Removing leading `/' from member names
2026-06-08T19:31:36.786643478Z INFO: HAProxy config collection complete.
2026-06-08T19:31:36.799239263Z INFO: 'previous-logs' folder not found on ip-10-0-132-184.ec2.internal, skipping...
2026-06-08T19:31:36.840119185Z error: the server doesn't have a resource type "machineconfigs"
2026-06-08T19:31:36.890681883Z INFO: Getting rules from prometheus-k8s-0
2026-06-08T19:31:37.007428269Z error: the server doesn't have a resource type "machineconfigpools"
2026-06-08T19:31:37.059349159Z Wrote inspect data to must-gather.
2026-06-08T19:31:37.088660603Z INFO: 'previous-logs' folder not found on ip-10-0-136-185.ec2.internal, skipping...
2026-06-08T19:31:37.088696346Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-06-08T19:31:37.088707350Z INFO: Machine Config Daemon termination log collection complete.
2026-06-08T19:31:37.231454199Z Wrote inspect data to must-gather.
2026-06-08T19:31:37.295255114Z INFO: Getting status/config from prometheus-k8s-0
2026-06-08T19:31:37.306382177Z Wrote inspect data to must-gather.
2026-06-08T19:31:37.392032264Z error: the server doesn't have a resource type "kubeletconfigs"
2026-06-08T19:31:37.559053136Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-06-08T19:31:37.595166699Z Wrote inspect data to must-gather.
2026-06-08T19:31:37.601304047Z INFO: INTERCONNECT MODE
2026-06-08T19:31:37.601419873Z INFO: Gathering ovn-kubernetes DBs
2026-06-08T19:31:37.733452407Z INFO: Getting status/flags from prometheus-k8s-0
2026-06-08T19:31:37.780451484Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:37.789787822Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-06-08T19:31:37.794995760Z INFO: Gathering OVN_Northbound from ovnkube-node-5wwrk...
2026-06-08T19:31:37.795195041Z INFO: Gathering OVN_Northbound from ovnkube-node-j9kf9...
2026-06-08T19:31:37.795436298Z INFO: Gathering OVN_Northbound from ovnkube-node-w6bfk...
2026-06-08T19:31:37.795722059Z INFO: Gathering OVN_Southbound from ovnkube-node-5wwrk...
2026-06-08T19:31:37.796190539Z INFO: Gathering OVN_Southbound from ovnkube-node-j9kf9...
2026-06-08T19:31:37.796666136Z INFO: Gathering OVN_Southbound from ovnkube-node-w6bfk...
2026-06-08T19:31:37.823773305Z Wrote inspect data to must-gather.
2026-06-08T19:31:38.090610546Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-06-08T19:31:38.144844725Z tar: Removing leading `/' from member names
2026-06-08T19:31:38.153703193Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:38.161853080Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-06-08T19:31:38.176091785Z tar: Removing leading `/' from member names
2026-06-08T19:31:38.177367427Z tar: Removing leading `/' from member names
2026-06-08T19:31:38.197596345Z tar: Removing leading `/' from member names
2026-06-08T19:31:38.255655855Z tar: Removing leading `/' from member names
2026-06-08T19:31:38.270429950Z Wrote inspect data to must-gather.
2026-06-08T19:31:38.304961693Z tar: Removing leading `/' from member names
2026-06-08T19:31:38.412052024Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:38.443766981Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-06-08T19:31:38.501910799Z W0608 19:31:38.501869    1422 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-06-08T19:31:38.508853458Z Wrote inspect data to must-gather.
2026-06-08T19:31:38.648670843Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-06-08T19:31:38.654938460Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:38.658641005Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-06-08T19:31:38.667825948Z INFO: Waiting for network log collection to complete ...
2026-06-08T19:31:38.667880690Z INFO: Waiting for ovnk database copies to complete ...
2026-06-08T19:31:38.668430774Z INFO: Copying ovnk databases complete.
2026-06-08T19:31:38.670527723Z 2.6M	must-gather/network_logs/ovnk_database_store
2026-06-08T19:31:38.673279858Z ovnk_database_store/
2026-06-08T19:31:38.673356600Z ovnk_database_store/ovnkube-node-5wwrk_nbdb
2026-06-08T19:31:38.682880610Z ovnk_database_store/ovnkube-node-5wwrk_sbdb
2026-06-08T19:31:38.704076726Z ovnk_database_store/ovnkube-node-j9kf9_sbdb
2026-06-08T19:31:38.723454975Z ovnk_database_store/ovnkube-node-w6bfk_nbdb
2026-06-08T19:31:38.731425685Z ovnk_database_store/ovnkube-node-j9kf9_nbdb
2026-06-08T19:31:38.739102797Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-06-08T19:31:38.739492020Z ovnk_database_store/ovnkube-node-w6bfk_sbdb
2026-06-08T19:31:38.757684207Z INFO: Network log collection complete.
2026-06-08T19:31:38.837647943Z Wrote inspect data to must-gather.
2026-06-08T19:31:38.848299321Z Gathering data for ns/openshift-network-console...
2026-06-08T19:31:38.876544798Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:39.091016746Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:39.094083181Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-06-08T19:31:39.137916169Z INFO: Getting status from alertmanager-main-0
2026-06-08T19:31:39.163598411Z Wrote inspect data to must-gather.
2026-06-08T19:31:39.354052447Z Gathering data for ns/openshift-console-operator...
2026-06-08T19:31:39.354143659Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:39.487020952Z Wrote inspect data to must-gather.
2026-06-08T19:31:39.551849731Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:39.554092901Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-06-08T19:31:39.729240230Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-06-08T19:31:39.729668378Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:39.731218846Z INFO: Fallback to identify the container image from release info
2026-06-08T19:31:39.788494943Z Gathering data for ns/openshift-console...
2026-06-08T19:31:39.918462644Z Wrote inspect data to /must-gather/istio.
2026-06-08T19:31:40.328040561Z Gathering data for ns/openshift-cluster-storage-operator...
2026-06-08T19:31:40.358010039Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75a8fe9618bb81a42d565c535f20bf55d78662baa7f32548859596b4c5aa793e
2026-06-08T19:31:40.503180948Z daemonset.apps/perf-node-gather-daemonset created
2026-06-08T19:31:40.558171417Z Done executing Istio gather script
2026-06-08T19:31:40.630666582Z Waiting for performance profile collector pods to become ready: 1
2026-06-08T19:31:40.866956560Z Gathering data for ns/openshift-dns-operator...
2026-06-08T19:31:41.171728663Z Gathering data for ns/openshift-dns...
2026-06-08T19:31:41.598567410Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T19:31:41.755993825Z Waiting for performance profile collector pods to become ready: 2
2026-06-08T19:31:41.773978511Z Gathering data for ns/openshift-image-registry...
2026-06-08T19:31:42.283098182Z Gathering data for ns/openshift-ingress-operator...
2026-06-08T19:31:42.582514975Z Gathering data for ns/openshift-ingress...
2026-06-08T19:31:42.878656730Z Waiting for performance profile collector pods to become ready: 3
2026-06-08T19:31:42.914766352Z Gathering data for ns/openshift-ingress-canary...
2026-06-08T19:31:43.272282356Z Gathering data for ns/openshift-insights...
2026-06-08T19:31:43.990761961Z Waiting for performance profile collector pods to become ready: 4
2026-06-08T19:31:45.097501292Z Waiting for performance profile collector pods to become ready: 5
2026-06-08T19:31:45.112749812Z Gathering data for ns/openshift-config...
2026-06-08T19:31:45.354858582Z Gathering data for ns/openshift-config-managed...
2026-06-08T19:31:45.741472048Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-06-08T19:31:45.997835283Z Gathering data for ns/openshift-kube-apiserver...
2026-06-08T19:31:46.218260330Z Waiting for performance profile collector pods to become ready: 6
2026-06-08T19:31:46.249592070Z Gathering data for ns/openshift-kube-controller-manager...
2026-06-08T19:31:46.517686721Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-06-08T19:31:46.603273066Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T19:31:46.797350051Z Gathering data for ns/openshift-kube-scheduler...
2026-06-08T19:31:47.077536465Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-06-08T19:31:47.343597762Z Waiting for performance profile collector pods to become ready: 7
2026-06-08T19:31:47.402593665Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-06-08T19:31:47.718517404Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-06-08T19:31:48.050122277Z Gathering data for ns/openshift-user-workload-monitoring...
2026-06-08T19:31:48.455368839Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-06-08T19:31:48.592114873Z Gathering data for ns/openshift-multus...
2026-06-08T19:31:48.690697812Z Collecting performance related data for node ip-10-0-136-185.ec2.internal
2026-06-08T19:31:48.697829298Z Collecting performance related data for node ip-10-0-128-82.ec2.internal
2026-06-08T19:31:48.705158218Z Collecting performance related data for node ip-10-0-132-184.ec2.internal
2026-06-08T19:31:50.264194214Z Gathering data for ns/openshift-ovn-kubernetes...
2026-06-08T19:31:51.572121561Z Gathering data for ns/openshift-host-network...
2026-06-08T19:31:51.609964396Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T19:31:51.919365560Z Gathering data for ns/openshift-network-diagnostics...
2026-06-08T19:31:52.465399451Z Gathering data for ns/openshift-network-node-identity...
2026-06-08T19:31:52.862342301Z Gathering data for ns/openshift-network-operator...
2026-06-08T19:31:53.258346886Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-06-08T19:31:53.668563103Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-06-08T19:31:54.215571734Z Gathering data for ns/openshift-apiserver-operator...
2026-06-08T19:31:54.513583739Z Gathering data for ns/openshift-apiserver...
2026-06-08T19:31:54.893135615Z Gathering data for ns/openshift-controller-manager-operator...
2026-06-08T19:31:55.150250089Z Gathering data for ns/openshift-controller-manager...
2026-06-08T19:31:55.411994183Z Gathering data for ns/openshift-cluster-samples-operator...
2026-06-08T19:31:56.049358244Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-06-08T19:31:56.327040158Z Gathering data for ns/openshift-service-ca-operator...
2026-06-08T19:31:56.619040436Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-06-08T19:31:56.634497527Z Gathering data for ns/openshift-service-ca...
2026-06-08T19:31:56.982558648Z Collecting kubelet logs for node ip-10-0-136-185.ec2.internal
2026-06-08T19:31:56.984468214Z Collecting kubelet logs for node ip-10-0-128-82.ec2.internal
2026-06-08T19:31:56.987206872Z Collecting kubelet logs for node ip-10-0-132-184.ec2.internal
2026-06-08T19:31:57.004253222Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-06-08T19:31:57.307466385Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-p7qhw namespace
2026-06-08T19:31:57.311894896Z INFO: Node performance data collection complete.
2026-06-08T19:31:57.892653794Z Wrote inspect data to must-gather.
2026-06-08T19:31:57.892701488Z error: inspection completed with the errors occurred while gathering data:
2026-06-08T19:31:57.892701488Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-06-08T19:31:58.011158411Z error: the server doesn't have a resource type "clusters"
2026-06-08T19:31:58.136555787Z Caches written to disk