2026-07-28T16:09:25.854172503Z [disk usage checker] Started
2026-07-28T16:09:25.857752672Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T16:09:26.053619827Z Gathering data for ns/openshift-cluster-version...
2026-07-28T16:09:26.189133146Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T16:09:26.278039161Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:09:26.360878822Z Gathering data for ns/default...
2026-07-28T16:09:26.693365199Z Gathering data for ns/openshift...
2026-07-28T16:09:27.119361703Z Gathering data for ns/kube-system...
2026-07-28T16:09:27.847499627Z Gathering data for ns/openshift-etcd...
2026-07-28T16:09:28.183717356Z Wrote inspect data to must-gather.
2026-07-28T16:09:28.183742417Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T16:09:28.183742417Z     namespaces "assisted-installer" not found
2026-07-28T16:09:28.566774026Z Waiting on subprocesses to finish execution.
2026-07-28T16:09:28.574532293Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T16:09:28.574532293Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T16:09:28.582122608Z INFO: Gathering HAProxy config files
2026-07-28T16:09:28.584849355Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T16:09:28.593989515Z Executing Istio gather script
2026-07-28T16:09:28.617456136Z INFO: Collecting host service logs for crio
2026-07-28T16:09:28.617776177Z INFO: Collecting host service logs for kubelet
2026-07-28T16:09:28.618132247Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T16:09:28.618464661Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T16:09:28.618797119Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T16:09:28.619168489Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T16:09:28.619492324Z INFO: Collecting host service logs for NetworkManager
2026-07-28T16:09:28.619802266Z INFO: Collecting host service logs for openvswitch
2026-07-28T16:09:28.620139228Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T16:09:28.620461318Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T16:09:28.620789317Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T16:09:28.621187414Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T16:09:28.710709260Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T16:09:29.174128306Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T16:09:29.316209963Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-lgx6t
2026-07-28T16:09:29.316261163Z insights-runtime-extractor-f5lm5
2026-07-28T16:09:29.316280200Z insights-runtime-extractor-kdn8k
2026-07-28T16:09:29.316298334Z insights-runtime-extractor-rsd4q
2026-07-28T16:09:29.484208696Z Wrote inspect data to must-gather.
2026-07-28T16:09:29.520613343Z No resources found in openshift-etcd namespace.
2026-07-28T16:09:29.608748826Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T16:09:29.608842482Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T16:09:29.719427847Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T16:09:29.836366485Z No resources found
2026-07-28T16:09:29.871132295Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T16:09:29.871132295Z See 'oc rsync -h' for help and examples
2026-07-28T16:09:29.909732967Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T16:09:29.917434812Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T16:09:29.977337551Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T16:09:30.462679681Z INFO: Worker host service log collection to complete.
2026-07-28T16:09:30.476203823Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T16:09:30.512470440Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T16:09:30.515331646Z ERROR: No running kube-apiserver pods found
2026-07-28T16:09:30.582289528Z Wrote inspect data to must-gather.
2026-07-28T16:09:30.830534427Z Gathering data for ns/openshift-monitoring...
2026-07-28T16:09:30.901859699Z Wrote inspect data to must-gather.
2026-07-28T16:09:30.949592160Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T16:09:31.080599917Z Wrote inspect data to must-gather.
2026-07-28T16:09:31.138682386Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T16:09:31.154388925Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T16:09:31.168228721Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:09:31.179468914Z No resources found
2026-07-28T16:09:31.186357911Z Wrote inspect data to must-gather.
2026-07-28T16:09:31.205860187Z INFO: 'previous-logs' folder not found on ip-10-0-131-121.ec2.internal, skipping...
2026-07-28T16:09:31.224170263Z Wrote inspect data to must-gather.
2026-07-28T16:09:31.331385258Z Wrote inspect data to must-gather.
2026-07-28T16:09:31.405258982Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T16:09:31.448223228Z tar: Removing leading `/' from member names
2026-07-28T16:09:31.453404736Z INFO: HAProxy config collection complete.
2026-07-28T16:09:31.508918115Z Wrote inspect data to must-gather.
2026-07-28T16:09:31.557217493Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T16:09:31.629920421Z INFO: 'previous-logs' folder not found on ip-10-0-133-16.ec2.internal, skipping...
2026-07-28T16:09:31.823920182Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T16:09:31.879043655Z Wrote inspect data to must-gather.
2026-07-28T16:09:31.946225956Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T16:09:31.984448395Z INFO: 'previous-logs' folder not found on ip-10-0-137-14.ec2.internal, skipping...
2026-07-28T16:09:31.984554370Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T16:09:31.984604987Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T16:09:32.071695027Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T16:09:32.095800772Z Wrote inspect data to must-gather.
2026-07-28T16:09:32.167120672Z Wrote inspect data to must-gather.
2026-07-28T16:09:32.263633726Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T16:09:32.373806163Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T16:09:32.487982484Z INFO: INTERCONNECT MODE
2026-07-28T16:09:32.488095376Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T16:09:32.526450865Z Wrote inspect data to must-gather.
2026-07-28T16:09:32.620741675Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T16:09:32.673349987Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T16:09:32.684826917Z INFO: Gathering OVN_Northbound from ovnkube-node-2vrz4...
2026-07-28T16:09:32.685142632Z INFO: Gathering OVN_Northbound from ovnkube-node-7hv4d...
2026-07-28T16:09:32.685358888Z INFO: Gathering OVN_Northbound from ovnkube-node-x5792...
2026-07-28T16:09:32.685648755Z INFO: Gathering OVN_Southbound from ovnkube-node-2vrz4...
2026-07-28T16:09:32.686075473Z INFO: Gathering OVN_Southbound from ovnkube-node-7hv4d...
2026-07-28T16:09:32.686575112Z INFO: Gathering OVN_Southbound from ovnkube-node-x5792...
2026-07-28T16:09:32.809937708Z Wrote inspect data to must-gather.
2026-07-28T16:09:32.993712944Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T16:09:33.028302011Z tar: Removing leading `/' from member names
2026-07-28T16:09:33.072225897Z tar: Removing leading `/' from member names
2026-07-28T16:09:33.079742633Z tar: Removing leading `/' from member names
2026-07-28T16:09:33.098129330Z tar: Removing leading `/' from member names
2026-07-28T16:09:33.102108785Z tar: Removing leading `/' from member names
2026-07-28T16:09:33.150487280Z tar: Removing leading `/' from member names
2026-07-28T16:09:33.242309814Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T16:09:33.256330162Z Wrote inspect data to must-gather.
2026-07-28T16:09:33.314636170Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T16:09:33.475535433Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:33.479272268Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T16:09:33.617263126Z Wrote inspect data to must-gather.
2026-07-28T16:09:33.623019287Z INFO: Waiting for network log collection to complete ...
2026-07-28T16:09:33.623151624Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T16:09:33.623618649Z INFO: Copying ovnk databases complete.
2026-07-28T16:09:33.625310982Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-28T16:09:33.627723057Z ovnk_database_store/
2026-07-28T16:09:33.627723057Z ovnk_database_store/ovnkube-node-2vrz4_nbdb
2026-07-28T16:09:33.637674206Z ovnk_database_store/ovnkube-node-x5792_nbdb
2026-07-28T16:09:33.646407478Z ovnk_database_store/ovnkube-node-7hv4d_sbdb
2026-07-28T16:09:33.672514619Z ovnk_database_store/ovnkube-node-2vrz4_sbdb
2026-07-28T16:09:33.697638296Z ovnk_database_store/ovnkube-node-7hv4d_nbdb
2026-07-28T16:09:33.708645198Z ovnk_database_store/ovnkube-node-x5792_sbdb
2026-07-28T16:09:33.711980224Z Gathering data for ns/openshift-network-console...
2026-07-28T16:09:33.717263115Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:33.719698630Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T16:09:33.732478983Z INFO: Network log collection complete.
2026-07-28T16:09:33.798975195Z INFO: Getting status from alertmanager-main-0
2026-07-28T16:09:33.955466932Z Wrote inspect data to must-gather.
2026-07-28T16:09:33.965343496Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:34.179217984Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:34.182315186Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T16:09:34.247086330Z Gathering data for ns/openshift-console-operator...
2026-07-28T16:09:34.253982468Z Wrote inspect data to must-gather.
2026-07-28T16:09:34.390750285Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:34.555637987Z Wrote inspect data to must-gather.
2026-07-28T16:09:34.601821136Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:34.605653117Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T16:09:34.684324028Z Gathering data for ns/openshift-console...
2026-07-28T16:09:34.850018089Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T16:09:34.853053970Z INFO: Fallback to identify the container image from release info
2026-07-28T16:09:34.864376342Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:35.059499616Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:35.063123334Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T16:09:35.251071595Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:35.270847979Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T16:09:35.451316289Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:35.453909218Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T16:09:35.459808417Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T16:09:35.601685497Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T16:09:35.659115677Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:35.761249802Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T16:09:35.858587174Z Gathering data for ns/openshift-dns-operator...
2026-07-28T16:09:35.880281504Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:35.883355768Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T16:09:35.958553466Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T16:09:36.063388892Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:36.225269154Z Gathering data for ns/openshift-dns...
2026-07-28T16:09:36.267318194Z Wrote inspect data to /must-gather/istio.
2026-07-28T16:09:36.910184749Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T16:09:36.911681347Z Done executing Istio gather script
2026-07-28T16:09:36.996512517Z Gathering data for ns/openshift-image-registry...
2026-07-28T16:09:37.458777295Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T16:09:37.826022907Z Gathering data for ns/openshift-ingress...
2026-07-28T16:09:38.019604210Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T16:09:38.154718189Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T16:09:38.533357534Z Gathering data for ns/openshift-insights...
2026-07-28T16:09:39.144824796Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T16:09:40.276959264Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T16:09:40.963223465Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T16:09:40.983028807Z Gathering data for ns/kserve...
2026-07-28T16:09:41.411853654Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T16:09:42.543174980Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T16:09:42.567062414Z Gathering data for ns/openshift-config...
2026-07-28T16:09:42.878155918Z Gathering data for ns/openshift-config-managed...
2026-07-28T16:09:43.296594349Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T16:09:43.557060180Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T16:09:43.677196454Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T16:09:43.898809004Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T16:09:43.943205743Z Collecting performance related data for node ip-10-0-137-14.ec2.internal
2026-07-28T16:09:43.951376128Z Collecting performance related data for node ip-10-0-133-16.ec2.internal
2026-07-28T16:09:43.962048461Z Collecting performance related data for node ip-10-0-131-121.ec2.internal
2026-07-28T16:09:44.233263531Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T16:09:44.623159792Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T16:09:44.941796613Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T16:09:45.343092653Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T16:09:45.735807842Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T16:09:45.970712400Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T16:09:46.031954560Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T16:09:46.599685211Z Gathering data for ns/openshift-multus...
2026-07-28T16:09:48.120931286Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T16:09:49.417470876Z Gathering data for ns/openshift-host-network...
2026-07-28T16:09:49.743063757Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T16:09:50.268714617Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T16:09:50.696487971Z Gathering data for ns/openshift-network-operator...
2026-07-28T16:09:50.975343464Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T16:09:51.082977810Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T16:09:51.444466364Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T16:09:51.955879390Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T16:09:52.046334961Z Collecting kubelet logs for node ip-10-0-137-14.ec2.internal
2026-07-28T16:09:52.048118552Z Collecting kubelet logs for node ip-10-0-133-16.ec2.internal
2026-07-28T16:09:52.049848515Z Collecting kubelet logs for node ip-10-0-131-121.ec2.internal
2026-07-28T16:09:52.309963759Z Gathering data for ns/openshift-apiserver...
2026-07-28T16:09:52.430813389Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-9zm2w namespace
2026-07-28T16:09:52.435843766Z INFO: Node performance data collection complete.
2026-07-28T16:09:52.814594962Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T16:09:53.117227008Z Gathering data for ns/openshift-controller-manager...
2026-07-28T16:09:53.383909511Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T16:09:54.017424854Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T16:09:54.299458673Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T16:09:54.620811793Z Gathering data for ns/openshift-service-ca...
2026-07-28T16:09:54.964656059Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T16:09:55.843709701Z Wrote inspect data to must-gather.
2026-07-28T16:09:55.843756217Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T16:09:55.843756217Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T16:09:55.968058560Z error: the server doesn't have a resource type "clusters"
2026-07-28T16:09:55.980126832Z [disk usage checker] Volume usage percentage: current = 11 ; allowed = 70
2026-07-28T16:09:56.111237198Z Caches written to disk