2026-07-28T07:01:49.500668902Z [disk usage checker] Started
2026-07-28T07:01:49.503920882Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T07:01:49.687095754Z Gathering data for ns/openshift-cluster-version...
2026-07-28T07:01:49.817471488Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T07:01:49.906961922Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T07:01:49.992020086Z Gathering data for ns/default...
2026-07-28T07:01:50.317040257Z Gathering data for ns/openshift...
2026-07-28T07:01:50.712137789Z Gathering data for ns/kube-system...
2026-07-28T07:01:51.391488762Z Gathering data for ns/openshift-etcd...
2026-07-28T07:01:51.680586680Z Wrote inspect data to must-gather.
2026-07-28T07:01:51.680617612Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T07:01:51.680617612Z     namespaces "assisted-installer" not found
2026-07-28T07:01:51.947022318Z Waiting on subprocesses to finish execution.
2026-07-28T07:01:51.956081544Z INFO: Gathering HAProxy config files
2026-07-28T07:01:51.962529654Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T07:01:51.966110529Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T07:01:51.966463275Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T07:01:51.968338303Z Executing Istio gather script
2026-07-28T07:01:51.970479753Z INFO: Collecting host service logs for crio
2026-07-28T07:01:51.970830444Z INFO: Collecting host service logs for kubelet
2026-07-28T07:01:51.971133450Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T07:01:51.971442729Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T07:01:51.971756380Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T07:01:51.972053184Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T07:01:51.972368420Z INFO: Collecting host service logs for NetworkManager
2026-07-28T07:01:51.972657618Z INFO: Collecting host service logs for openvswitch
2026-07-28T07:01:51.972946804Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T07:01:51.973244453Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T07:01:51.973551369Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T07:01:51.973944048Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T07:01:52.012526372Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T07:01:52.315906408Z No resources found
2026-07-28T07:01:52.344079348Z No resources found in openshift-etcd namespace.
2026-07-28T07:01:52.497635446Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T07:01:52.671509156Z Wrote inspect data to must-gather.
2026-07-28T07:01:52.909821535Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T07:01:52.909852929Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T07:01:53.088066724Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T07:01:53.168653132Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T07:01:53.186966611Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T07:01:53.360007452Z INFO: Worker host service log collection to complete.
2026-07-28T07:01:53.363034736Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T07:01:53.393964326Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T07:01:53.425940148Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-pz549
2026-07-28T07:01:53.425940148Z insights-runtime-extractor-599tf
2026-07-28T07:01:53.425940148Z insights-runtime-extractor-99dbc
2026-07-28T07:01:53.425940148Z insights-runtime-extractor-tvk2j
2026-07-28T07:01:53.677016924Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T07:01:53.742258573Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T07:01:54.008836985Z ERROR: No running kube-apiserver pods found
2026-07-28T07:01:54.062946146Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.086873796Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.089695999Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T07:01:54.094778180Z No resources found
2026-07-28T07:01:54.125594691Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.200109510Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.206953524Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.211181455Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T07:01:54.211181455Z See 'oc rsync -h' for help and examples
2026-07-28T07:01:54.227801859Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T07:01:54.242137650Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.327574587Z INFO: 'previous-logs' folder not found on ip-10-0-130-140.ec2.internal, skipping...
2026-07-28T07:01:54.422440323Z tar: Removing leading `/' from member names
2026-07-28T07:01:54.433587051Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T07:01:54.434242601Z INFO: HAProxy config collection complete.
2026-07-28T07:01:54.513845874Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T07:01:54.550060837Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.576405256Z Gathering data for ns/openshift-monitoring...
2026-07-28T07:01:54.638594454Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T07:01:54.653786097Z INFO: 'previous-logs' folder not found on ip-10-0-136-217.ec2.internal, skipping...
2026-07-28T07:01:54.710652235Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T07:01:54.845206755Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.914607723Z Wrote inspect data to must-gather.
2026-07-28T07:01:54.936282682Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T07:01:54.939056335Z INFO: 'previous-logs' folder not found on ip-10-0-139-74.ec2.internal, skipping...
2026-07-28T07:01:54.939078122Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T07:01:54.939099284Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T07:01:55.084383690Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T07:01:55.126560241Z Wrote inspect data to must-gather.
2026-07-28T07:01:55.128482537Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T07:01:55.294342427Z Wrote inspect data to must-gather.
2026-07-28T07:01:55.408951496Z INFO: INTERCONNECT MODE
2026-07-28T07:01:55.409040393Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T07:01:55.505967134Z Wrote inspect data to must-gather.
2026-07-28T07:01:55.525090600Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T07:01:55.550573042Z INFO: Gathering OVN_Northbound from ovnkube-node-hnzst...
2026-07-28T07:01:55.550816315Z INFO: Gathering OVN_Northbound from ovnkube-node-l6vbb...
2026-07-28T07:01:55.551056297Z INFO: Gathering OVN_Northbound from ovnkube-node-lhsp6...
2026-07-28T07:01:55.551289764Z INFO: Gathering OVN_Southbound from ovnkube-node-hnzst...
2026-07-28T07:01:55.551536495Z INFO: Gathering OVN_Southbound from ovnkube-node-l6vbb...
2026-07-28T07:01:55.551806404Z INFO: Gathering OVN_Southbound from ovnkube-node-lhsp6...
2026-07-28T07:01:55.837258580Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T07:01:55.854594562Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T07:01:55.858604853Z tar: Removing leading `/' from member names
2026-07-28T07:01:55.868206705Z tar: Removing leading `/' from member names
2026-07-28T07:01:55.880827014Z tar: Removing leading `/' from member names
2026-07-28T07:01:55.885216559Z Wrote inspect data to must-gather.
2026-07-28T07:01:55.906013571Z tar: Removing leading `/' from member names
2026-07-28T07:01:55.906013571Z tar: Removing leading `/' from member names
2026-07-28T07:01:55.912498592Z tar: Removing leading `/' from member names
2026-07-28T07:01:56.087581635Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:56.133234329Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T07:01:56.143174830Z Wrote inspect data to must-gather.
2026-07-28T07:01:56.180558398Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T07:01:56.331636274Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:56.334202891Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T07:01:56.372998320Z Wrote inspect data to must-gather.
2026-07-28T07:01:56.376290025Z INFO: Waiting for network log collection to complete ...
2026-07-28T07:01:56.376342123Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T07:01:56.376730060Z INFO: Copying ovnk databases complete.
2026-07-28T07:01:56.378160050Z 2.7M	must-gather/network_logs/ovnk_database_store
2026-07-28T07:01:56.380690901Z ovnk_database_store/
2026-07-28T07:01:56.380690901Z ovnk_database_store/ovnkube-node-lhsp6_nbdb
2026-07-28T07:01:56.387528283Z ovnk_database_store/ovnkube-node-lhsp6_sbdb
2026-07-28T07:01:56.392020296Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T07:01:56.404119640Z ovnk_database_store/ovnkube-node-l6vbb_sbdb
2026-07-28T07:01:56.423518197Z ovnk_database_store/ovnkube-node-hnzst_sbdb
2026-07-28T07:01:56.442652746Z ovnk_database_store/ovnkube-node-hnzst_nbdb
2026-07-28T07:01:56.449743758Z ovnk_database_store/ovnkube-node-l6vbb_nbdb
2026-07-28T07:01:56.461025675Z INFO: Network log collection complete.
2026-07-28T07:01:56.544487098Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:56.576412236Z Wrote inspect data to must-gather.
2026-07-28T07:01:56.748352675Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:56.750779347Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T07:01:56.753836483Z INFO: Getting status from alertmanager-main-0
2026-07-28T07:01:56.788373403Z W0728 07:01:56.788309    1541 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T07:01:56.796172193Z Wrote inspect data to must-gather.
2026-07-28T07:01:56.826047212Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T07:01:56.960833124Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:57.012219194Z Gathering data for ns/openshift-network-console...
2026-07-28T07:01:57.057642844Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T07:01:57.060851902Z INFO: Fallback to identify the container image from release info
2026-07-28T07:01:57.170484280Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:57.173081959Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T07:01:57.380182712Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:57.421507826Z Gathering data for ns/openshift-console-operator...
2026-07-28T07:01:57.569206318Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:57.571662880Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T07:01:57.612018317Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T07:01:57.740510385Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T07:01:57.744492826Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:57.818019981Z Gathering data for ns/openshift-console...
2026-07-28T07:01:57.876109465Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T07:01:57.952457515Z Wrote inspect data to /must-gather/istio.
2026-07-28T07:01:58.368343707Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T07:01:58.593812574Z Done executing Istio gather script
2026-07-28T07:01:58.854348206Z Gathering data for ns/openshift-dns-operator...
2026-07-28T07:01:58.990193365Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T07:01:59.138195522Z Gathering data for ns/openshift-dns...
2026-07-28T07:01:59.519032972Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T07:01:59.769820552Z Gathering data for ns/openshift-image-registry...
2026-07-28T07:02:00.110092591Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T07:02:00.227527056Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T07:02:00.533069415Z Gathering data for ns/openshift-ingress...
2026-07-28T07:02:00.872291938Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T07:02:01.232874556Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T07:02:01.252642027Z Gathering data for ns/openshift-insights...
2026-07-28T07:02:02.336086432Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T07:02:02.990831451Z Gathering data for ns/openshift-jobset-operator...
2026-07-28T07:02:03.440184275Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T07:02:03.739323434Z Gathering data for ns/openshift-config...
2026-07-28T07:02:03.976405563Z Gathering data for ns/openshift-config-managed...
2026-07-28T07:02:04.431767147Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T07:02:04.523716684Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T07:02:04.559585066Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T07:02:04.701005979Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T07:02:05.007417349Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T07:02:05.280763615Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T07:02:05.539754624Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T07:02:05.668888776Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T07:02:05.796708289Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T07:02:05.916743587Z Collecting performance related data for node ip-10-0-136-217.ec2.internal
2026-07-28T07:02:05.924007493Z Collecting performance related data for node ip-10-0-130-140.ec2.internal
2026-07-28T07:02:05.935435739Z Collecting performance related data for node ip-10-0-139-74.ec2.internal
2026-07-28T07:02:06.119989378Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T07:02:06.483585688Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T07:02:06.820732187Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T07:02:07.427676691Z Gathering data for ns/openshift-multus...
2026-07-28T07:02:08.894058075Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T07:02:09.531261390Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T07:02:10.062663899Z Gathering data for ns/openshift-host-network...
2026-07-28T07:02:10.382859782Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T07:02:10.901317175Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T07:02:11.299593373Z Gathering data for ns/openshift-network-operator...
2026-07-28T07:02:11.658450496Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T07:02:11.903591595Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T07:02:12.338598559Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T07:02:12.612352303Z Gathering data for ns/openshift-apiserver...
2026-07-28T07:02:12.972229062Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T07:02:13.217852061Z Gathering data for ns/openshift-controller-manager...
2026-07-28T07:02:13.477330105Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T07:02:13.803089060Z Collecting kubelet logs for node ip-10-0-136-217.ec2.internal
2026-07-28T07:02:13.804823122Z Collecting kubelet logs for node ip-10-0-130-140.ec2.internal
2026-07-28T07:02:13.806585412Z Collecting kubelet logs for node ip-10-0-139-74.ec2.internal
2026-07-28T07:02:14.126961755Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-8qw9c namespace
2026-07-28T07:02:14.133317572Z INFO: Node performance data collection complete.
2026-07-28T07:02:14.136141405Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T07:02:14.423523597Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T07:02:14.535976784Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-28T07:02:14.717699863Z Gathering data for ns/openshift-service-ca...
2026-07-28T07:02:15.058964629Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T07:02:15.890376654Z Wrote inspect data to must-gather.
2026-07-28T07:02:15.890422470Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T07:02:15.890422470Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T07:02:16.009577997Z error: the server doesn't have a resource type "clusters"
2026-07-28T07:02:16.165483788Z Caches written to disk