2026-09-17T12:23:14.987552802Z [disk usage checker] Started
2026-09-17T12:23:14.990755041Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-17T12:23:15.186398061Z Gathering data for ns/openshift-cluster-version...
2026-09-17T12:23:15.310878597Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-17T12:23:15.398648215Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-17T12:23:15.480205249Z Gathering data for ns/default...
2026-09-17T12:23:15.794919910Z Gathering data for ns/openshift...
2026-09-17T12:23:16.195059629Z Gathering data for ns/kube-system...
2026-09-17T12:23:16.867516584Z Gathering data for ns/openshift-etcd...
2026-09-17T12:23:17.164861346Z Wrote inspect data to must-gather.
2026-09-17T12:23:17.164894834Z error: inspection completed with the errors occurred while gathering data:
2026-09-17T12:23:17.164894834Z     namespaces "assisted-installer" not found
2026-09-17T12:23:17.539024663Z Waiting on subprocesses to finish execution.
2026-09-17T12:23:17.546957129Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-09-17T12:23:17.547644496Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-09-17T12:23:17.553531490Z INFO: Gathering machine config daemon's old logs from all nodes
2026-09-17T12:23:17.559238022Z INFO: Collecting host service logs for crio
2026-09-17T12:23:17.559515893Z INFO: Collecting host service logs for kubelet
2026-09-17T12:23:17.559887993Z INFO: Collecting host service logs for rpm-ostreed
2026-09-17T12:23:17.560284675Z INFO: Collecting host service logs for ostree-finalize-staged
2026-09-17T12:23:17.560587947Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-09-17T12:23:17.560889661Z INFO: Collecting host service logs for machine-config-daemon-host
2026-09-17T12:23:17.561183713Z INFO: Collecting host service logs for NetworkManager
2026-09-17T12:23:17.561476509Z INFO: Collecting host service logs for openvswitch
2026-09-17T12:23:17.561768066Z INFO: Collecting host service logs for ovs-configuration
2026-09-17T12:23:17.562121416Z INFO: Collecting host service logs for ovsdb-server
2026-09-17T12:23:17.562420472Z INFO: Collecting host service logs for ovs-vswitchd
2026-09-17T12:23:17.562782611Z INFO: Waiting for worker host service log collection to complete ...
2026-09-17T12:23:17.567181419Z INFO: Gathering HAProxy config files
2026-09-17T12:23:17.579913108Z Executing Istio gather script
2026-09-17T12:23:17.813665634Z INFO: Waiting for node performance related collection to complete ...
2026-09-17T12:23:18.362669744Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-09-17T12:23:18.439233449Z No resources found
2026-09-17T12:23:18.445360390Z INFO: "sriov-network-operator" not detected. Skipping.
2026-09-17T12:23:18.472388714Z No resources found in openshift-etcd namespace.
2026-09-17T12:23:18.526988867Z error: the server doesn't have a resource type "performanceprofile"
2026-09-17T12:23:18.621980011Z INFO: Found 1 replicas - prometheus-k8s-0
2026-09-17T12:23:19.064254556Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-09-17T12:23:19.373273828Z Wrote inspect data to must-gather.
2026-09-17T12:23:19.450254089Z Wrote inspect data to must-gather.
2026-09-17T12:23:19.506654549Z Wrote inspect data to must-gather.
2026-09-17T12:23:19.514976797Z INFO: "metallb-operator" not detected. Skipping.
2026-09-17T12:23:19.515262932Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-09-17T12:23:19.518337820Z Wrote inspect data to must-gather.
2026-09-17T12:23:19.536311178Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-09-17T12:23:19.536341030Z INFO: on-disk MachineConfig config collection complete.
2026-09-17T12:23:19.582557363Z Wrote inspect data to must-gather.
2026-09-17T12:23:19.589146346Z INFO: Collecting Insights Archives from insights-operator-5844b856db-g7q5c
2026-09-17T12:23:19.589197387Z insights-runtime-extractor-q4rs2
2026-09-17T12:23:19.589220602Z insights-runtime-extractor-szm4d
2026-09-17T12:23:19.589242715Z insights-runtime-extractor-xcptd
2026-09-17T12:23:19.619836749Z Wrote inspect data to must-gather.
2026-09-17T12:23:19.733303257Z No resources found
2026-09-17T12:23:19.758326998Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-17T12:23:19.825725689Z Wrote inspect data to must-gather.
2026-09-17T12:23:19.838960650Z INFO: Worker host service log collection to complete.
2026-09-17T12:23:19.869664570Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-09-17T12:23:19.919575458Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-09-17T12:23:19.919575458Z See 'oc rsync -h' for help and examples
2026-09-17T12:23:19.928310112Z ERROR: No running kube-apiserver pods found
2026-09-17T12:23:19.950890079Z error: the server doesn't have a resource type "machineconfigs"
2026-09-17T12:23:20.000335183Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-17T12:23:20.044746685Z INFO: 'previous-logs' folder not found on ip-10-0-134-202.ec2.internal, skipping...
2026-09-17T12:23:20.158890179Z INFO: Waiting for HAProxy config collection to complete ...
2026-09-17T12:23:20.190933524Z Wrote inspect data to must-gather.
2026-09-17T12:23:20.195391644Z error: the server doesn't have a resource type "machineconfigpools"
2026-09-17T12:23:20.317689120Z Gathering data for ns/openshift-monitoring...
2026-09-17T12:23:20.435785931Z INFO: Getting rules from prometheus-k8s-0
2026-09-17T12:23:20.442977340Z tar: Removing leading `/' from member names
2026-09-17T12:23:20.444929522Z INFO: 'previous-logs' folder not found on ip-10-0-139-126.ec2.internal, skipping...
2026-09-17T12:23:20.448148395Z INFO: HAProxy config collection complete.
2026-09-17T12:23:20.496409237Z Wrote inspect data to must-gather.
2026-09-17T12:23:20.537961555Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-09-17T12:23:20.546806767Z Wrote inspect data to must-gather.
2026-09-17T12:23:20.697279257Z error: the server doesn't have a resource type "kubeletconfigs"
2026-09-17T12:23:20.729395398Z INFO: 'previous-logs' folder not found on ip-10-0-142-23.ec2.internal, skipping...
2026-09-17T12:23:20.729432975Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-09-17T12:23:20.729453492Z INFO: Machine Config Daemon termination log collection complete.
2026-09-17T12:23:20.815870618Z Wrote inspect data to must-gather.
2026-09-17T12:23:20.858493264Z INFO: Getting status/config from prometheus-k8s-0
2026-09-17T12:23:20.916827849Z Wrote inspect data to must-gather.
2026-09-17T12:23:21.100043039Z INFO: INTERCONNECT MODE
2026-09-17T12:23:21.100142536Z INFO: Gathering ovn-kubernetes DBs
2026-09-17T12:23:21.125530259Z Wrote inspect data to must-gather.
2026-09-17T12:23:21.253464977Z INFO: Getting status/flags from prometheus-k8s-0
2026-09-17T12:23:21.259348622Z INFO: Gathering OVN_Northbound from ovnkube-node-9kddg...
2026-09-17T12:23:21.259595424Z INFO: Gathering OVN_Northbound from ovnkube-node-fq9nr...
2026-09-17T12:23:21.259893994Z INFO: Gathering OVN_Northbound from ovnkube-node-vr9br...
2026-09-17T12:23:21.260179826Z INFO: Gathering OVN_Southbound from ovnkube-node-9kddg...
2026-09-17T12:23:21.260530076Z INFO: Gathering OVN_Southbound from ovnkube-node-fq9nr...
2026-09-17T12:23:21.260844757Z INFO: Gathering OVN_Southbound from ovnkube-node-vr9br...
2026-09-17T12:23:21.296337743Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-09-17T12:23:21.384304018Z Wrote inspect data to must-gather.
2026-09-17T12:23:21.559261522Z tar: Removing leading `/' from member names
2026-09-17T12:23:21.578498207Z tar: Removing leading `/' from member names
2026-09-17T12:23:21.579467967Z tar: Removing leading `/' from member names
2026-09-17T12:23:21.581902370Z tar: Removing leading `/' from member names
2026-09-17T12:23:21.612039531Z tar: Removing leading `/' from member names
2026-09-17T12:23:21.633683374Z tar: Removing leading `/' from member names
2026-09-17T12:23:21.635793928Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-09-17T12:23:21.650629373Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:21.670080834Z Wrote inspect data to must-gather.
2026-09-17T12:23:21.895485791Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-09-17T12:23:21.902083474Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:21.905225457Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-09-17T12:23:21.927689951Z Wrote inspect data to must-gather.
2026-09-17T12:23:21.945704121Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-09-17T12:23:22.080477007Z INFO: Waiting for network log collection to complete ...
2026-09-17T12:23:22.080526041Z INFO: Waiting for ovnk database copies to complete ...
2026-09-17T12:23:22.081181302Z INFO: Copying ovnk databases complete.
2026-09-17T12:23:22.083081141Z 3.6M	must-gather/network_logs/ovnk_database_store
2026-09-17T12:23:22.087280311Z ovnk_database_store/
2026-09-17T12:23:22.087333699Z ovnk_database_store/ovnkube-node-fq9nr_sbdb
2026-09-17T12:23:22.104568502Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:22.108276323Z ovnk_database_store/ovnkube-node-9kddg_sbdb
2026-09-17T12:23:22.129304120Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-09-17T12:23:22.133478028Z ovnk_database_store/ovnkube-node-fq9nr_nbdb
2026-09-17T12:23:22.142727650Z ovnk_database_store/ovnkube-node-vr9br_sbdb
2026-09-17T12:23:22.164976848Z Wrote inspect data to must-gather.
2026-09-17T12:23:22.166669649Z ovnk_database_store/ovnkube-node-vr9br_nbdb
2026-09-17T12:23:22.175412708Z ovnk_database_store/ovnkube-node-9kddg_nbdb
2026-09-17T12:23:22.188122900Z INFO: Network log collection complete.
2026-09-17T12:23:22.317854052Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:22.319894524Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-09-17T12:23:22.370093384Z Wrote inspect data to must-gather.
2026-09-17T12:23:22.511261215Z INFO: Getting status from alertmanager-main-0
2026-09-17T12:23:22.522032149Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:22.582141897Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-09-17T12:23:22.642906453Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-09-17T12:23:22.645250332Z INFO: Fallback to identify the container image from release info
2026-09-17T12:23:22.743127718Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:22.746685010Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-09-17T12:23:22.760913513Z Gathering data for ns/openshift-network-console...
2026-09-17T12:23:22.952478316Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:23.151262060Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:23.153994384Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-09-17T12:23:23.160292459Z Gathering data for ns/openshift-console-operator...
2026-09-17T12:23:23.315502876Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:23.384090422Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a
2026-09-17T12:23:23.513358701Z daemonset.apps/perf-node-gather-daemonset created
2026-09-17T12:23:23.523920095Z Wrote inspect data to /must-gather/istio.
2026-09-17T12:23:23.553208160Z Gathering data for ns/openshift-console...
2026-09-17T12:23:23.642515694Z Waiting for performance profile collector pods to become ready: 1
2026-09-17T12:23:24.095886168Z Gathering data for ns/openshift-cluster-storage-operator...
2026-09-17T12:23:24.223744431Z Done executing Istio gather script
2026-09-17T12:23:24.556106973Z Gathering data for ns/openshift-dns-operator...
2026-09-17T12:23:24.758593160Z Waiting for performance profile collector pods to become ready: 2
2026-09-17T12:23:24.822904272Z Gathering data for ns/openshift-dns...
2026-09-17T12:23:25.005465750Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-17T12:23:25.386087775Z Gathering data for ns/openshift-image-registry...
2026-09-17T12:23:25.838323769Z Gathering data for ns/openshift-ingress-operator...
2026-09-17T12:23:25.884417841Z Waiting for performance profile collector pods to become ready: 3
2026-09-17T12:23:26.155264577Z Gathering data for ns/openshift-ingress...
2026-09-17T12:23:26.453385292Z Gathering data for ns/openshift-ingress-canary...
2026-09-17T12:23:26.820184738Z Gathering data for ns/openshift-insights...
2026-09-17T12:23:26.985568561Z Waiting for performance profile collector pods to become ready: 4
2026-09-17T12:23:28.101252528Z Waiting for performance profile collector pods to become ready: 5
2026-09-17T12:23:28.814037886Z Gathering data for ns/openshift-config...
2026-09-17T12:23:29.070908984Z Gathering data for ns/openshift-config-managed...
2026-09-17T12:23:29.217762979Z Waiting for performance profile collector pods to become ready: 6
2026-09-17T12:23:29.488287501Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-09-17T12:23:29.740340933Z Gathering data for ns/openshift-kube-apiserver...
2026-09-17T12:23:29.979057246Z Gathering data for ns/openshift-kube-controller-manager...
2026-09-17T12:23:30.010885677Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-17T12:23:30.247155599Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-09-17T12:23:30.343456749Z Waiting for performance profile collector pods to become ready: 7
2026-09-17T12:23:30.518238361Z Gathering data for ns/openshift-kube-scheduler...
2026-09-17T12:23:30.760961471Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-09-17T12:23:31.029970517Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-09-17T12:23:31.336074016Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-09-17T12:23:31.461696674Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-09-17T12:23:31.642960973Z Gathering data for ns/openshift-user-workload-monitoring...
2026-09-17T12:23:31.716276064Z Collecting performance related data for node ip-10-0-139-126.ec2.internal
2026-09-17T12:23:31.723321058Z Collecting performance related data for node ip-10-0-142-23.ec2.internal
2026-09-17T12:23:31.729993290Z Collecting performance related data for node ip-10-0-134-202.ec2.internal
2026-09-17T12:23:32.197653708Z Gathering data for ns/openshift-multus...
2026-09-17T12:23:33.700200745Z Gathering data for ns/openshift-ovn-kubernetes...
2026-09-17T12:23:34.895140036Z Gathering data for ns/openshift-host-network...
2026-09-17T12:23:35.016894005Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-17T12:23:35.187192635Z Gathering data for ns/openshift-network-diagnostics...
2026-09-17T12:23:35.697904702Z Gathering data for ns/openshift-network-node-identity...
2026-09-17T12:23:36.145777457Z Gathering data for ns/openshift-network-operator...
2026-09-17T12:23:36.518802921Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-09-17T12:23:36.826131049Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-09-17T12:23:37.336933197Z Gathering data for ns/openshift-apiserver-operator...
2026-09-17T12:23:37.612436455Z Gathering data for ns/openshift-apiserver...
2026-09-17T12:23:37.974014389Z Gathering data for ns/openshift-controller-manager-operator...
2026-09-17T12:23:38.226167975Z Gathering data for ns/openshift-controller-manager...
2026-09-17T12:23:38.508143549Z Gathering data for ns/openshift-cluster-samples-operator...
2026-09-17T12:23:39.180910690Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-09-17T12:23:39.444485209Z Collecting kubelet logs for node ip-10-0-139-126.ec2.internal
2026-09-17T12:23:39.446324229Z Collecting kubelet logs for node ip-10-0-142-23.ec2.internal
2026-09-17T12:23:39.448478649Z Collecting kubelet logs for node ip-10-0-134-202.ec2.internal
2026-09-17T12:23:39.469363481Z Gathering data for ns/openshift-service-ca-operator...
2026-09-17T12:23:39.815234514Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-f59bn namespace
2026-09-17T12:23:39.820520148Z INFO: Node performance data collection complete.
2026-09-17T12:23:39.825157900Z Gathering data for ns/openshift-service-ca...
2026-09-17T12:23:40.021767655Z [disk usage checker] Volume usage percentage: current = 10 ; allowed = 70
2026-09-17T12:23:40.205509156Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-09-17T12:23:41.044198799Z Wrote inspect data to must-gather.
2026-09-17T12:23:41.044250276Z error: inspection completed with the errors occurred while gathering data:
2026-09-17T12:23:41.044250276Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-09-17T12:23:41.170496920Z error: the server doesn't have a resource type "clusters"
2026-09-17T12:23:41.341475377Z Caches written to disk