2026-09-16T12:22:53.816614285Z [disk usage checker] Started
2026-09-16T12:22:53.820089301Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-16T12:22:54.015691486Z Gathering data for ns/openshift-cluster-version...
2026-09-16T12:22:54.143073738Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T12:22:54.232323832Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T12:22:54.311327064Z Gathering data for ns/default...
2026-09-16T12:22:54.596628208Z Gathering data for ns/openshift...
2026-09-16T12:22:54.981954651Z Gathering data for ns/kube-system...
2026-09-16T12:22:55.722661586Z Gathering data for ns/openshift-etcd...
2026-09-16T12:22:56.008539551Z Wrote inspect data to must-gather.
2026-09-16T12:22:56.008581268Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T12:22:56.008581268Z     namespaces "assisted-installer" not found
2026-09-16T12:22:56.453349162Z Waiting on subprocesses to finish execution.
2026-09-16T12:22:56.460722169Z Executing Istio gather script
2026-09-16T12:22:56.463016369Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-16T12:22:56.466512598Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-16T12:22:56.470467317Z INFO: Gathering HAProxy config files
2026-09-16T12:22:56.471031695Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-16T12:22:56.476663251Z INFO: Collecting host service logs for crio
2026-09-16T12:22:56.476953959Z INFO: Collecting host service logs for kubelet
2026-09-16T12:22:56.477302022Z INFO: Collecting host service logs for rpm-ostreed
2026-09-16T12:22:56.477678275Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-16T12:22:56.478037488Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-16T12:22:56.478366679Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-16T12:22:56.478753907Z INFO: Collecting host service logs for NetworkManager
2026-09-16T12:22:56.479125110Z INFO: Collecting host service logs for openvswitch
2026-09-16T12:22:56.479477000Z INFO: Collecting host service logs for ovs-configuration
2026-09-16T12:22:56.479850779Z INFO: Collecting host service logs for ovsdb-server
2026-09-16T12:22:56.480207645Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-16T12:22:56.480629516Z INFO: Waiting for worker host service log collection to complete ...
2026-09-16T12:22:56.541214218Z INFO: Waiting for node performance related collection to complete ...
2026-09-16T12:22:57.525720993Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-16T12:22:57.555190399Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-16T12:22:57.592193548Z Wrote inspect data to must-gather.
2026-09-16T12:22:57.593379355Z No resources found
2026-09-16T12:22:57.628907315Z INFO: Collecting Insights Archives from insights-operator-5844b856db-7c5bm
2026-09-16T12:22:57.628907315Z insights-runtime-extractor-2dh2h
2026-09-16T12:22:57.628907315Z insights-runtime-extractor-6mk2z
2026-09-16T12:22:57.628907315Z insights-runtime-extractor-krb84
2026-09-16T12:22:57.659667572Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-16T12:22:57.659717531Z INFO: on-disk MachineConfig config collection complete.
2026-09-16T12:22:57.772029277Z No resources found in openshift-etcd namespace.
2026-09-16T12:22:57.994070942Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-16T12:22:58.000790935Z INFO: "metallb-operator" not detected. Skipping.
2026-09-16T12:22:58.025268958Z error: the server doesn't have a resource type "performanceprofile"
2026-09-16T12:22:58.091109966Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-16T12:22:58.438150961Z INFO: Worker host service log collection to complete.
2026-09-16T12:22:58.604339666Z Wrote inspect data to must-gather.
2026-09-16T12:22:58.728189584Z Wrote inspect data to must-gather.
2026-09-16T12:22:58.742627781Z No resources found
2026-09-16T12:22:58.747199661Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T12:22:58.812530361Z Wrote inspect data to must-gather.
2026-09-16T12:22:58.829433637Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-16T12:22:58.904210610Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-16T12:22:58.911666009Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-16T12:22:58.911666009Z See 'oc rsync -h' for help and examples
2026-09-16T12:22:58.927226300Z Wrote inspect data to must-gather.
2026-09-16T12:22:58.931411294Z ERROR: No running kube-apiserver pods found
2026-09-16T12:22:58.941100187Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-16T12:22:58.941524981Z Wrote inspect data to must-gather.
2026-09-16T12:22:59.000972577Z Wrote inspect data to must-gather.
2026-09-16T12:22:59.016959724Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-16T12:22:59.034927708Z Gathering data for ns/openshift-monitoring...
2026-09-16T12:22:59.095466194Z INFO: 'previous-logs' folder not found on ip-10-0-131-122.ec2.internal, skipping...
2026-09-16T12:22:59.276457852Z error: the server doesn't have a resource type "machineconfigs"
2026-09-16T12:22:59.287851957Z tar: Removing leading `/' from member names
2026-09-16T12:22:59.292925119Z INFO: HAProxy config collection complete.
2026-09-16T12:22:59.322871461Z Wrote inspect data to must-gather.
2026-09-16T12:22:59.342664294Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-16T12:22:59.421408902Z INFO: 'previous-logs' folder not found on ip-10-0-140-32.ec2.internal, skipping...
2026-09-16T12:22:59.463637988Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-16T12:22:59.518611393Z INFO: Getting rules from prometheus-k8s-0
2026-09-16T12:22:59.615990847Z Wrote inspect data to must-gather.
2026-09-16T12:22:59.711112019Z Wrote inspect data to must-gather.
2026-09-16T12:22:59.723915321Z INFO: 'previous-logs' folder not found on ip-10-0-143-53.ec2.internal, skipping...
2026-09-16T12:22:59.723958889Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-16T12:22:59.723983917Z INFO: Machine Config Daemon termination log collection complete.
2026-09-16T12:22:59.886818972Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-16T12:22:59.894309178Z Wrote inspect data to must-gather.
2026-09-16T12:22:59.952336369Z INFO: Getting status/config from prometheus-k8s-0
2026-09-16T12:23:00.109383033Z Wrote inspect data to must-gather.
2026-09-16T12:23:00.179477392Z INFO: INTERCONNECT MODE
2026-09-16T12:23:00.179544760Z INFO: Gathering ovn-kubernetes DBs
2026-09-16T12:23:00.249681759Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-16T12:23:00.342014252Z Wrote inspect data to must-gather.
2026-09-16T12:23:00.346162236Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-16T12:23:00.358859509Z INFO: Gathering OVN_Northbound from ovnkube-node-rmwwg...
2026-09-16T12:23:00.359190220Z INFO: Gathering OVN_Northbound from ovnkube-node-vrlt6...
2026-09-16T12:23:00.359441163Z INFO: Gathering OVN_Northbound from ovnkube-node-wjpll...
2026-09-16T12:23:00.359810180Z INFO: Gathering OVN_Southbound from ovnkube-node-rmwwg...
2026-09-16T12:23:00.360132707Z INFO: Gathering OVN_Southbound from ovnkube-node-vrlt6...
2026-09-16T12:23:00.360431219Z INFO: Gathering OVN_Southbound from ovnkube-node-wjpll...
2026-09-16T12:23:00.533106217Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:00.651398841Z tar: Removing leading `/' from member names
2026-09-16T12:23:00.654899566Z tar: Removing leading `/' from member names
2026-09-16T12:23:00.687531477Z tar: Removing leading `/' from member names
2026-09-16T12:23:00.694706982Z Wrote inspect data to must-gather.
2026-09-16T12:23:00.712942291Z tar: Removing leading `/' from member names
2026-09-16T12:23:00.748950081Z tar: Removing leading `/' from member names
2026-09-16T12:23:00.753425406Z tar: Removing leading `/' from member names
2026-09-16T12:23:00.781664693Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-16T12:23:00.794853652Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-16T12:23:00.862976112Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:00.865804669Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-16T12:23:00.952178513Z Wrote inspect data to must-gather.
2026-09-16T12:23:01.078923794Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-16T12:23:01.120306972Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:01.219089479Z Wrote inspect data to must-gather.
2026-09-16T12:23:01.261276564Z INFO: Waiting for network log collection to complete ...
2026-09-16T12:23:01.261396772Z INFO: Waiting for ovnk database copies to complete ...
2026-09-16T12:23:01.261926856Z INFO: Copying ovnk databases complete.
2026-09-16T12:23:01.265449535Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-09-16T12:23:01.267865076Z ovnk_database_store/
2026-09-16T12:23:01.267865076Z ovnk_database_store/ovnkube-node-vrlt6_sbdb
2026-09-16T12:23:01.292989980Z ovnk_database_store/ovnkube-node-wjpll_nbdb
2026-09-16T12:23:01.302012915Z ovnk_database_store/ovnkube-node-vrlt6_nbdb
2026-09-16T12:23:01.310661314Z ovnk_database_store/ovnkube-node-rmwwg_nbdb
2026-09-16T12:23:01.320272483Z ovnk_database_store/ovnkube-node-rmwwg_sbdb
2026-09-16T12:23:01.346076046Z ovnk_database_store/ovnkube-node-wjpll_sbdb
2026-09-16T12:23:01.348659833Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:01.351984102Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-16T12:23:01.356341401Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-16T12:23:01.374430215Z INFO: Network log collection complete.
2026-09-16T12:23:01.467164936Z W0916 12:23:01.467107    1499 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-09-16T12:23:01.474636801Z Wrote inspect data to must-gather.
2026-09-16T12:23:01.484052905Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-16T12:23:01.569665028Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:01.665411779Z Gathering data for ns/openshift-network-console...
2026-09-16T12:23:01.791846932Z INFO: Getting status from alertmanager-main-0
2026-09-16T12:23:01.793893954Z Wrote inspect data to must-gather.
2026-09-16T12:23:01.803322523Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:01.806372872Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-16T12:23:02.053898453Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:02.077355327Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-16T12:23:02.079696454Z INFO: Fallback to identify the container image from release info
2026-09-16T12:23:02.122802575Z Gathering data for ns/openshift-console-operator...
2026-09-16T12:23:02.266116759Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:02.268327910Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-16T12:23:02.429246574Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:02.533053466Z Gathering data for ns/openshift-console...
2026-09-16T12:23:02.639952108Z Wrote inspect data to /must-gather/istio.
2026-09-16T12:23:02.707992733Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-16T12:23:02.844436945Z daemonset.apps/perf-node-gather-daemonset created
2026-09-16T12:23:02.990971147Z Waiting for performance profile collector pods to become ready: 1
2026-09-16T12:23:03.088904134Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-16T12:23:03.355679096Z Done executing Istio gather script
2026-09-16T12:23:03.601405403Z Gathering data for ns/openshift-dns-operator...
2026-09-16T12:23:03.891045017Z Gathering data for ns/openshift-dns...
2026-09-16T12:23:03.913970813Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-16T12:23:04.096340272Z Waiting for performance profile collector pods to become ready: 2
2026-09-16T12:23:04.427836011Z Gathering data for ns/openshift-image-registry...
2026-09-16T12:23:04.864554924Z Gathering data for ns/openshift-ingress-operator...
2026-09-16T12:23:05.162626358Z Gathering data for ns/openshift-ingress...
2026-09-16T12:23:05.223682872Z Waiting for performance profile collector pods to become ready: 3
2026-09-16T12:23:05.464485253Z Gathering data for ns/openshift-ingress-canary...
2026-09-16T12:23:05.813531978Z Gathering data for ns/openshift-insights...
2026-09-16T12:23:06.345936772Z Waiting for performance profile collector pods to become ready: 4
2026-09-16T12:23:07.462141532Z Waiting for performance profile collector pods to become ready: 5
2026-09-16T12:23:07.792711231Z Gathering data for ns/openshift-config...
2026-09-16T12:23:08.083678091Z Gathering data for ns/openshift-config-managed...
2026-09-16T12:23:08.499325454Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-16T12:23:08.590988182Z Waiting for performance profile collector pods to become ready: 6
2026-09-16T12:23:08.797347532Z Gathering data for ns/openshift-kube-apiserver...
2026-09-16T12:23:08.918670459Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-16T12:23:09.065769055Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-16T12:23:09.353218826Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-16T12:23:09.680981167Z Gathering data for ns/openshift-kube-scheduler...
2026-09-16T12:23:09.724849757Z Waiting for performance profile collector pods to become ready: 7
2026-09-16T12:23:09.933916450Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-16T12:23:10.197047310Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-16T12:23:10.499131475Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-16T12:23:10.819737111Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-16T12:23:10.861245346Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-16T12:23:11.120279845Z Collecting performance related data for node ip-10-0-140-32.ec2.internal
2026-09-16T12:23:11.130132383Z Collecting performance related data for node ip-10-0-131-122.ec2.internal
2026-09-16T12:23:11.138274860Z Collecting performance related data for node ip-10-0-143-53.ec2.internal
2026-09-16T12:23:11.445756200Z Gathering data for ns/openshift-multus...
2026-09-16T12:23:12.906331303Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-16T12:23:13.925976434Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-16T12:23:14.081421867Z Gathering data for ns/openshift-host-network...
2026-09-16T12:23:14.367713279Z Gathering data for ns/openshift-network-diagnostics...
2026-09-16T12:23:14.876534807Z Gathering data for ns/openshift-network-node-identity...
2026-09-16T12:23:15.260030937Z Gathering data for ns/openshift-network-operator...
2026-09-16T12:23:15.629003090Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-16T12:23:15.873912151Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-16T12:23:16.343435908Z Gathering data for ns/openshift-apiserver-operator...
2026-09-16T12:23:16.632846473Z Gathering data for ns/openshift-apiserver...
2026-09-16T12:23:17.014892561Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-16T12:23:17.278950732Z Gathering data for ns/openshift-controller-manager...
2026-09-16T12:23:17.552743263Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-16T12:23:18.208731138Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-16T12:23:18.481557683Z Gathering data for ns/openshift-service-ca-operator...
2026-09-16T12:23:18.760654683Z Gathering data for ns/openshift-service-ca...
2026-09-16T12:23:18.932806977Z [disk usage checker] Volume usage percentage: current = 16 ; allowed = 70
2026-09-16T12:23:19.118676144Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-16T12:23:19.732598305Z Collecting kubelet logs for node ip-10-0-140-32.ec2.internal
2026-09-16T12:23:19.734389573Z Collecting kubelet logs for node ip-10-0-131-122.ec2.internal
2026-09-16T12:23:19.736683547Z Collecting kubelet logs for node ip-10-0-143-53.ec2.internal
2026-09-16T12:23:20.049429794Z Wrote inspect data to must-gather.
2026-09-16T12:23:20.049689886Z error: inspection completed with the errors occurred while gathering data:
2026-09-16T12:23:20.049689886Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-16T12:23:20.089221730Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-mjh2m namespace
2026-09-16T12:23:20.095136394Z INFO: Node performance data collection complete.
2026-09-16T12:23:20.226969514Z error: the server doesn't have a resource type "clusters"
2026-09-16T12:23:20.354998916Z Caches written to disk