2026-09-15T17:21:45.260081461Z [disk usage checker] Started
2026-09-15T17:21:45.263566237Z [disk usage checker] Volume usage percentage: current = 48 ; allowed = 70
2026-09-15T17:21:45.645447207Z Gathering data for ns/openshift-cluster-version...
2026-09-15T17:21:45.776237425Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-15T17:21:45.867362164Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-15T17:21:45.944168338Z Gathering data for ns/default...
2026-09-15T17:21:46.235920767Z Gathering data for ns/openshift...
2026-09-15T17:21:46.672208851Z Gathering data for ns/kube-system...
2026-09-15T17:21:47.410719966Z Gathering data for ns/openshift-etcd...
2026-09-15T17:21:47.732612091Z Wrote inspect data to must-gather.
2026-09-15T17:21:47.732642623Z error: inspection completed with the errors occurred while gathering data:
2026-09-15T17:21:47.732642623Z     namespaces "assisted-installer" not found
2026-09-15T17:21:48.280337896Z Waiting on subprocesses to finish execution.
2026-09-15T17:21:48.287698154Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-15T17:21:48.289662542Z Executing Istio gather script
2026-09-15T17:21:48.290343561Z INFO: Collecting host service logs for crio
2026-09-15T17:21:48.290648506Z INFO: Collecting host service logs for kubelet
2026-09-15T17:21:48.291079190Z INFO: Collecting host service logs for rpm-ostreed
2026-09-15T17:21:48.291410076Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-15T17:21:48.291719492Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-15T17:21:48.292042207Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-15T17:21:48.292332953Z INFO: Collecting host service logs for NetworkManager
2026-09-15T17:21:48.292621243Z INFO: Collecting host service logs for openvswitch
2026-09-15T17:21:48.293037524Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-15T17:21:48.293364604Z INFO: Collecting host service logs for ovs-configuration
2026-09-15T17:21:48.293676863Z INFO: Collecting host service logs for ovsdb-server
2026-09-15T17:21:48.294005064Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-15T17:21:48.294353573Z INFO: Waiting for worker host service log collection to complete ...
2026-09-15T17:21:48.304117426Z INFO: Gathering HAProxy config files
2026-09-15T17:21:48.326866854Z INFO: Waiting for node performance related collection to complete ...
2026-09-15T17:21:48.344641226Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-15T17:21:48.751317368Z No resources found in openshift-etcd namespace.
2026-09-15T17:21:48.980132039Z error: the server doesn't have a resource type "performanceprofile"
2026-09-15T17:21:48.984546095Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-15T17:21:49.038976622Z No resources found
2026-09-15T17:21:49.438216921Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-15T17:21:49.525198963Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-15T17:21:49.622238710Z INFO: "metallb-operator" not detected. Skipping.
2026-09-15T17:21:49.624930805Z INFO: Collecting Insights Archives from insights-operator-5844b856db-nhpth
2026-09-15T17:21:49.624967482Z insights-runtime-extractor-8vhhh
2026-09-15T17:21:49.624987759Z insights-runtime-extractor-ffhfm
2026-09-15T17:21:49.625006630Z insights-runtime-extractor-l6547
2026-09-15T17:21:49.661732417Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-15T17:21:49.805887705Z Wrote inspect data to must-gather.
2026-09-15T17:21:49.955895749Z INFO: Worker host service log collection to complete.
2026-09-15T17:21:49.971649548Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-15T17:21:49.981163131Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-15T17:21:49.981163131Z INFO: on-disk MachineConfig config collection complete.
2026-09-15T17:21:50.057962915Z Wrote inspect data to must-gather.
2026-09-15T17:21:50.101474983Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-15T17:21:50.101474983Z See 'oc rsync -h' for help and examples
2026-09-15T17:21:50.205221833Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-15T17:21:50.307387255Z [disk usage checker] Volume usage percentage: current = 48 ; allowed = 70
2026-09-15T17:21:50.307862483Z Wrote inspect data to must-gather.
2026-09-15T17:21:50.347219106Z ERROR: No running kube-apiserver pods found
2026-09-15T17:21:50.624517276Z tar: Removing leading `/' from member names
2026-09-15T17:21:50.632090677Z INFO: HAProxy config collection complete.
2026-09-15T17:21:50.752264070Z No resources found
2026-09-15T17:21:50.784149854Z Gathering data for ns/openshift-monitoring...
2026-09-15T17:21:50.792697544Z INFO: 'previous-logs' folder not found on ip-10-0-130-184.ec2.internal, skipping...
2026-09-15T17:21:50.956400606Z Wrote inspect data to must-gather.
2026-09-15T17:21:50.976686084Z error: the server doesn't have a resource type "machineconfigs"
2026-09-15T17:21:51.118307533Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-15T17:21:51.119104903Z Wrote inspect data to must-gather.
2026-09-15T17:21:51.126965551Z Wrote inspect data to must-gather.
2026-09-15T17:21:51.184241833Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-15T17:21:51.257777635Z Wrote inspect data to must-gather.
2026-09-15T17:21:51.280830932Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-15T17:21:51.341414456Z INFO: 'previous-logs' folder not found on ip-10-0-132-85.ec2.internal, skipping...
2026-09-15T17:21:51.374945869Z INFO: Getting rules from prometheus-k8s-0
2026-09-15T17:21:51.579181287Z Wrote inspect data to must-gather.
2026-09-15T17:21:51.587657830Z Wrote inspect data to must-gather.
2026-09-15T17:21:51.664697548Z INFO: 'previous-logs' folder not found on ip-10-0-141-27.ec2.internal, skipping...
2026-09-15T17:21:51.664809618Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-15T17:21:51.664840695Z INFO: Machine Config Daemon termination log collection complete.
2026-09-15T17:21:51.778777313Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-15T17:21:51.875256106Z INFO: Getting status/config from prometheus-k8s-0
2026-09-15T17:21:51.885047965Z Wrote inspect data to must-gather.
2026-09-15T17:21:52.045535150Z Wrote inspect data to must-gather.
2026-09-15T17:21:52.095777641Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-15T17:21:52.167505446Z Wrote inspect data to must-gather.
2026-09-15T17:21:52.332002427Z Wrote inspect data to must-gather.
2026-09-15T17:21:52.369324422Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-15T17:21:52.399607578Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-15T17:21:52.536960105Z INFO: INTERCONNECT MODE
2026-09-15T17:21:52.537003624Z INFO: Gathering ovn-kubernetes DBs
2026-09-15T17:21:52.605939839Z Wrote inspect data to must-gather.
2026-09-15T17:21:52.666549765Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-15T17:21:52.731276884Z INFO: Gathering OVN_Northbound from ovnkube-node-n9ptp...
2026-09-15T17:21:52.731483184Z INFO: Gathering OVN_Northbound from ovnkube-node-qtgkt...
2026-09-15T17:21:52.731707815Z INFO: Gathering OVN_Northbound from ovnkube-node-sc5tl...
2026-09-15T17:21:52.732067571Z INFO: Gathering OVN_Southbound from ovnkube-node-n9ptp...
2026-09-15T17:21:52.732437438Z INFO: Gathering OVN_Southbound from ovnkube-node-qtgkt...
2026-09-15T17:21:52.732772683Z INFO: Gathering OVN_Southbound from ovnkube-node-sc5tl...
2026-09-15T17:21:52.952232800Z Wrote inspect data to must-gather.
2026-09-15T17:21:53.043472883Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-15T17:21:53.084609518Z tar: Removing leading `/' from member names
2026-09-15T17:21:53.105505615Z tar: Removing leading `/' from member names
2026-09-15T17:21:53.117866507Z tar: Removing leading `/' from member names
2026-09-15T17:21:53.137536490Z tar: Removing leading `/' from member names
2026-09-15T17:21:53.152210282Z tar: Removing leading `/' from member names
2026-09-15T17:21:53.157785656Z tar: Removing leading `/' from member names
2026-09-15T17:21:53.191817731Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-15T17:21:53.211429633Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-15T17:21:53.377530577Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-15T17:21:53.415919860Z Wrote inspect data to must-gather.
2026-09-15T17:21:53.437491552Z Gathering data for ns/openshift-network-console...
2026-09-15T17:21:53.446223025Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:53.716827558Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:53.722178746Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-15T17:21:53.765644161Z INFO: Waiting for network log collection to complete ...
2026-09-15T17:21:53.765793434Z INFO: Waiting for ovnk database copies to complete ...
2026-09-15T17:21:53.766356576Z INFO: Copying ovnk databases complete.
2026-09-15T17:21:53.768457942Z 4.7M	must-gather/network_logs/ovnk_database_store
2026-09-15T17:21:53.771330136Z ovnk_database_store/
2026-09-15T17:21:53.771636307Z ovnk_database_store/ovnkube-node-n9ptp_nbdb
2026-09-15T17:21:53.785321849Z W0915 17:21:53.785277    1499 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-15T17:21:53.786260280Z ovnk_database_store/ovnkube-node-n9ptp_sbdb
2026-09-15T17:21:53.800992997Z Wrote inspect data to must-gather.
2026-09-15T17:21:53.810681323Z ovnk_database_store/ovnkube-node-sc5tl_nbdb
2026-09-15T17:21:53.823939318Z ovnk_database_store/ovnkube-node-qtgkt_nbdb
2026-09-15T17:21:53.838595279Z ovnk_database_store/ovnkube-node-qtgkt_sbdb
2026-09-15T17:21:53.866412772Z INFO: Getting status from alertmanager-main-0
2026-09-15T17:21:53.871678461Z ovnk_database_store/ovnkube-node-sc5tl_sbdb
2026-09-15T17:21:53.906587946Z INFO: Network log collection complete.
2026-09-15T17:21:53.978984818Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:54.038124131Z Gathering data for ns/openshift-console-operator...
2026-09-15T17:21:54.167859326Z Wrote inspect data to must-gather.
2026-09-15T17:21:54.215728976Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:54.218944927Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-15T17:21:54.441670677Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:54.457626452Z Gathering data for ns/openshift-console...
2026-09-15T17:21:54.463515456Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-15T17:21:54.466991745Z INFO: Fallback to identify the container image from release info
2026-09-15T17:21:54.652153514Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:54.655394750Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-15T17:21:54.873784747Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:54.996614701Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-15T17:21:55.093532192Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:55.098413822Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-09-15T17:21:55.155625145Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-15T17:21:55.296659629Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:55.305523718Z daemonset.apps/perf-node-gather-daemonset created
2026-09-15T17:21:55.326445063Z [disk usage checker] Volume usage percentage: current = 48 ; allowed = 70
2026-09-15T17:21:55.466064947Z Waiting for performance profile collector pods to become ready: 1
2026-09-15T17:21:55.529442634Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:55.533825968Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-09-15T17:21:55.581340390Z Gathering data for ns/openshift-dns-operator...
2026-09-15T17:21:55.714322513Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:55.909541371Z Gathering data for ns/openshift-dns...
2026-09-15T17:21:55.934789710Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:55.937874319Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-15T17:21:56.093112013Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:56.287467612Z Wrote inspect data to /must-gather/istio.
2026-09-15T17:21:56.549717032Z Gathering data for ns/openshift-image-registry...
2026-09-15T17:21:56.610843262Z Waiting for performance profile collector pods to become ready: 2
2026-09-15T17:21:57.020174527Z Done executing Istio gather script
2026-09-15T17:21:57.041765375Z Gathering data for ns/openshift-ingress-operator...
2026-09-15T17:21:57.350519889Z Gathering data for ns/openshift-ingress...
2026-09-15T17:21:57.670054921Z Gathering data for ns/openshift-ingress-canary...
2026-09-15T17:21:57.737824743Z Waiting for performance profile collector pods to become ready: 3
2026-09-15T17:21:58.038423931Z Gathering data for ns/openshift-insights...
2026-09-15T17:21:58.856808503Z Waiting for performance profile collector pods to become ready: 4
2026-09-15T17:21:59.975497799Z Waiting for performance profile collector pods to become ready: 5
2026-09-15T17:22:00.331557624Z [disk usage checker] Volume usage percentage: current = 48 ; allowed = 70
2026-09-15T17:22:00.465693656Z Gathering data for ns/kserve...
2026-09-15T17:22:01.091070063Z Waiting for performance profile collector pods to become ready: 6
2026-09-15T17:22:02.224818673Z Waiting for performance profile collector pods to become ready: 7
2026-09-15T17:22:02.577404481Z Gathering data for ns/openshift-config...
2026-09-15T17:22:02.840677113Z Gathering data for ns/openshift-config-managed...
2026-09-15T17:22:03.248860657Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-15T17:22:03.365458958Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-15T17:22:03.525819326Z Gathering data for ns/openshift-kube-apiserver...
2026-09-15T17:22:03.655510825Z Collecting performance related data for node ip-10-0-132-85.ec2.internal
2026-09-15T17:22:03.664686880Z Collecting performance related data for node ip-10-0-130-184.ec2.internal
2026-09-15T17:22:03.675265795Z Collecting performance related data for node ip-10-0-141-27.ec2.internal
2026-09-15T17:22:03.833274237Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-15T17:22:04.152950827Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-15T17:22:04.515536976Z Gathering data for ns/openshift-kube-scheduler...
2026-09-15T17:22:04.843464688Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-15T17:22:05.200116755Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-15T17:22:05.338471376Z [disk usage checker] Volume usage percentage: current = 48 ; allowed = 70
2026-09-15T17:22:05.552516525Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-15T17:22:05.867935516Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-15T17:22:06.415346617Z Gathering data for ns/openshift-multus...
2026-09-15T17:22:07.840941563Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-15T17:22:09.106118722Z Gathering data for ns/openshift-host-network...
2026-09-15T17:22:09.443278732Z Gathering data for ns/openshift-network-diagnostics...
2026-09-15T17:22:09.946842280Z Gathering data for ns/openshift-network-node-identity...
2026-09-15T17:22:10.343979422Z [disk usage checker] Volume usage percentage: current = 48 ; allowed = 70
2026-09-15T17:22:10.359860662Z Gathering data for ns/openshift-network-operator...
2026-09-15T17:22:10.728649682Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-15T17:22:10.989371766Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-15T17:22:11.450418978Z Gathering data for ns/openshift-apiserver-operator...
2026-09-15T17:22:11.726551539Z Gathering data for ns/openshift-apiserver...
2026-09-15T17:22:12.108514977Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-15T17:22:12.354363028Z Gathering data for ns/openshift-controller-manager...
2026-09-15T17:22:12.403207682Z Collecting kubelet logs for node ip-10-0-132-85.ec2.internal
2026-09-15T17:22:12.405718679Z Collecting kubelet logs for node ip-10-0-130-184.ec2.internal
2026-09-15T17:22:12.408077283Z Collecting kubelet logs for node ip-10-0-141-27.ec2.internal
2026-09-15T17:22:12.673901580Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-15T17:22:12.879352536Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-87zwh namespace
2026-09-15T17:22:12.885106402Z INFO: Node performance data collection complete.
2026-09-15T17:22:13.346619736Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-15T17:22:13.614602102Z Gathering data for ns/openshift-service-ca-operator...
2026-09-15T17:22:13.906389041Z Gathering data for ns/openshift-service-ca...
2026-09-15T17:22:14.238609411Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-15T17:22:15.093458533Z Wrote inspect data to must-gather.
2026-09-15T17:22:15.093503553Z error: inspection completed with the errors occurred while gathering data:
2026-09-15T17:22:15.093503553Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-15T17:22:15.224922193Z error: the server doesn't have a resource type "clusters"
2026-09-15T17:22:15.349896791Z [disk usage checker] Volume usage percentage: current = 48 ; allowed = 70
2026-09-15T17:22:15.396220055Z Caches written to disk