2026-07-01T17:25:21.334545698Z [disk usage checker] Started
2026-07-01T17:25:21.337905847Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-01T17:25:21.543594505Z Gathering data for ns/openshift-cluster-version...
2026-07-01T17:25:21.684069521Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-01T17:25:21.775939382Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-01T17:25:21.857489620Z Gathering data for ns/default...
2026-07-01T17:25:22.173649197Z Gathering data for ns/openshift...
2026-07-01T17:25:22.619421749Z Gathering data for ns/kube-system...
2026-07-01T17:25:23.426447797Z Gathering data for ns/openshift-etcd...
2026-07-01T17:25:23.743066209Z Wrote inspect data to must-gather.
2026-07-01T17:25:23.743093241Z error: inspection completed with the errors occurred while gathering data:
2026-07-01T17:25:23.743093241Z     namespaces "assisted-installer" not found
2026-07-01T17:25:24.133640403Z Waiting on subprocesses to finish execution.
2026-07-01T17:25:24.138547901Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-01T17:25:24.144184800Z Executing Istio gather script
2026-07-01T17:25:24.146956987Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-01T17:25:24.149382952Z INFO: Gathering HAProxy config files
2026-07-01T17:25:24.153379425Z INFO: Collecting host service logs for crio
2026-07-01T17:25:24.153745237Z INFO: Collecting host service logs for kubelet
2026-07-01T17:25:24.154093405Z INFO: Collecting host service logs for rpm-ostreed
2026-07-01T17:25:24.154423477Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-01T17:25:24.154812168Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-01T17:25:24.155153974Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-01T17:25:24.155496326Z INFO: Collecting host service logs for NetworkManager
2026-07-01T17:25:24.155839886Z INFO: Collecting host service logs for openvswitch
2026-07-01T17:25:24.156189649Z INFO: Collecting host service logs for ovs-configuration
2026-07-01T17:25:24.156548259Z INFO: Collecting host service logs for ovsdb-server
2026-07-01T17:25:24.156910562Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-01T17:25:24.157313571Z INFO: Waiting for worker host service log collection to complete ...
2026-07-01T17:25:24.161698662Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-01T17:25:24.182583019Z INFO: Waiting for node performance related collection to complete ...
2026-07-01T17:25:25.155804665Z No resources found
2026-07-01T17:25:25.216375005Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-01T17:25:25.348277867Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-01T17:25:25.348308848Z INFO: on-disk MachineConfig config collection complete.
2026-07-01T17:25:25.380673253Z error: the server doesn't have a resource type "performanceprofile"
2026-07-01T17:25:25.417928724Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-01T17:25:25.437912224Z Wrote inspect data to must-gather.
2026-07-01T17:25:25.447875681Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-fssvm
2026-07-01T17:25:25.447917264Z insights-runtime-extractor-4sdv9
2026-07-01T17:25:25.447938744Z insights-runtime-extractor-8ww9j
2026-07-01T17:25:25.447959679Z insights-runtime-extractor-v5999
2026-07-01T17:25:25.450766255Z INFO: "metallb-operator" not detected. Skipping.
2026-07-01T17:25:25.615277752Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-01T17:25:25.687558955Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-01T17:25:25.691824123Z No resources found in openshift-etcd namespace.
2026-07-01T17:25:25.694661356Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-01T17:25:25.729323409Z INFO: Worker host service log collection to complete.
2026-07-01T17:25:25.853356864Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-01T17:25:25.853356864Z See 'oc rsync -h' for help and examples
2026-07-01T17:25:26.062415861Z ERROR: No running kube-apiserver pods found
2026-07-01T17:25:26.116067404Z Wrote inspect data to must-gather.
2026-07-01T17:25:26.130197585Z Wrote inspect data to must-gather.
2026-07-01T17:25:26.348465023Z tar: Removing leading `/' from member names
2026-07-01T17:25:26.352326894Z INFO: HAProxy config collection complete.
2026-07-01T17:25:26.405058782Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-01T17:25:26.427654091Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-01T17:25:26.470047694Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-01T17:25:26.725242537Z Wrote inspect data to must-gather.
2026-07-01T17:25:26.726134630Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-01T17:25:26.750030155Z Wrote inspect data to must-gather.
2026-07-01T17:25:26.789547548Z No resources found
2026-07-01T17:25:26.879953461Z Wrote inspect data to must-gather.
2026-07-01T17:25:26.896264292Z INFO: 'previous-logs' folder not found on ip-10-0-137-244.ec2.internal, skipping...
2026-07-01T17:25:26.961338426Z error: the server doesn't have a resource type "machineconfigs"
2026-07-01T17:25:27.005651059Z Wrote inspect data to must-gather.
2026-07-01T17:25:27.156698186Z Wrote inspect data to must-gather.
2026-07-01T17:25:27.174181316Z Gathering data for ns/openshift-monitoring...
2026-07-01T17:25:27.203377156Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-01T17:25:27.284401807Z INFO: 'previous-logs' folder not found on ip-10-0-138-194.ec2.internal, skipping...
2026-07-01T17:25:27.368336946Z INFO: Getting rules from prometheus-k8s-0
2026-07-01T17:25:27.449979215Z Wrote inspect data to must-gather.
2026-07-01T17:25:27.500362413Z Wrote inspect data to must-gather.
2026-07-01T17:25:27.594236268Z INFO: 'previous-logs' folder not found on ip-10-0-139-133.ec2.internal, skipping...
2026-07-01T17:25:27.594325203Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-01T17:25:27.594376562Z INFO: Machine Config Daemon termination log collection complete.
2026-07-01T17:25:27.701415441Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-01T17:25:27.754706495Z Wrote inspect data to must-gather.
2026-07-01T17:25:27.800704624Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-01T17:25:27.878207327Z INFO: Getting status/config from prometheus-k8s-0
2026-07-01T17:25:27.946869147Z Wrote inspect data to must-gather.
2026-07-01T17:25:28.087136569Z INFO: INTERCONNECT MODE
2026-07-01T17:25:28.087163818Z INFO: Gathering ovn-kubernetes DBs
2026-07-01T17:25:28.181467694Z Wrote inspect data to must-gather.
2026-07-01T17:25:28.277487895Z INFO: Gathering OVN_Northbound from ovnkube-node-8qjcb...
2026-07-01T17:25:28.277837541Z INFO: Gathering OVN_Northbound from ovnkube-node-dtdwc...
2026-07-01T17:25:28.278110487Z INFO: Gathering OVN_Northbound from ovnkube-node-zjsw4...
2026-07-01T17:25:28.278414335Z INFO: Gathering OVN_Southbound from ovnkube-node-8qjcb...
2026-07-01T17:25:28.278698056Z INFO: Gathering OVN_Southbound from ovnkube-node-dtdwc...
2026-07-01T17:25:28.278962736Z INFO: Gathering OVN_Southbound from ovnkube-node-zjsw4...
2026-07-01T17:25:28.419646639Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-01T17:25:28.519814650Z Wrote inspect data to must-gather.
2026-07-01T17:25:28.580138440Z tar: Removing leading `/' from member names
2026-07-01T17:25:28.634231039Z tar: Removing leading `/' from member names
2026-07-01T17:25:28.647453565Z tar: Removing leading `/' from member names
2026-07-01T17:25:28.679757257Z tar: Removing leading `/' from member names
2026-07-01T17:25:28.685530946Z tar: Removing leading `/' from member names
2026-07-01T17:25:28.715053930Z tar: Removing leading `/' from member names
2026-07-01T17:25:28.772284634Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-01T17:25:28.792556549Z Wrote inspect data to must-gather.
2026-07-01T17:25:28.854540239Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-01T17:25:29.069728344Z Wrote inspect data to must-gather.
2026-07-01T17:25:29.070265244Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-01T17:25:29.084551952Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-01T17:25:29.117804612Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:29.188261873Z INFO: Waiting for network log collection to complete ...
2026-07-01T17:25:29.188375271Z INFO: Waiting for ovnk database copies to complete ...
2026-07-01T17:25:29.188916337Z INFO: Copying ovnk databases complete.
2026-07-01T17:25:29.190963995Z 3.4M	must-gather/network_logs/ovnk_database_store
2026-07-01T17:25:29.193844398Z ovnk_database_store/
2026-07-01T17:25:29.193890161Z ovnk_database_store/ovnkube-node-8qjcb_nbdb
2026-07-01T17:25:29.208766856Z ovnk_database_store/ovnkube-node-zjsw4_sbdb
2026-07-01T17:25:29.230262177Z ovnk_database_store/ovnkube-node-zjsw4_nbdb
2026-07-01T17:25:29.240513273Z ovnk_database_store/ovnkube-node-8qjcb_sbdb
2026-07-01T17:25:29.259888015Z ovnk_database_store/ovnkube-node-dtdwc_nbdb
2026-07-01T17:25:29.268084828Z ovnk_database_store/ovnkube-node-dtdwc_sbdb
2026-07-01T17:25:29.291437022Z INFO: Network log collection complete.
2026-07-01T17:25:29.342988417Z Wrote inspect data to must-gather.
2026-07-01T17:25:29.375888287Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-01T17:25:29.393407277Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:29.396083181Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-01T17:25:29.593456764Z Wrote inspect data to must-gather.
2026-07-01T17:25:29.631023594Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:29.783918961Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-01T17:25:29.819408438Z INFO: Getting status from alertmanager-main-0
2026-07-01T17:25:29.866473787Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:29.871006082Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-01T17:25:29.922873025Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-01T17:25:29.926246815Z INFO: Fallback to identify the container image from release info
2026-07-01T17:25:29.990635285Z Gathering data for ns/openshift-network-console...
2026-07-01T17:25:30.094105262Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:30.299809138Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:30.302607742Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-01T17:25:30.466116199Z Gathering data for ns/openshift-console-operator...
2026-07-01T17:25:30.536655058Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:30.736780038Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:30.738967441Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io
2026-07-01T17:25:30.923405416Z Gathering data for ns/openshift-console...
2026-07-01T17:25:30.929551576Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:31.128590179Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:31.131098982Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io
2026-07-01T17:25:31.296876336Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:31.434585528Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-01T17:25:31.506195162Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:31.509202827Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io
2026-07-01T17:25:31.521829251Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-01T17:25:31.689606538Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:31.894716765Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:31.898116076Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-01T17:25:32.088733583Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:32.097361382Z Gathering data for ns/openshift-dns-operator...
2026-07-01T17:25:32.286061363Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:32.289173366Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-01T17:25:32.429594897Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f
2026-07-01T17:25:32.437038216Z Gathering data for ns/openshift-dns...
2026-07-01T17:25:32.491008865Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:32.562580411Z daemonset.apps/perf-node-gather-daemonset created
2026-07-01T17:25:32.692060229Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:32.696634988Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-01T17:25:32.703016001Z Waiting for performance profile collector pods to become ready: 1
2026-07-01T17:25:32.865529624Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:33.105606607Z Wrote inspect data to /must-gather/istio.
2026-07-01T17:25:33.179080349Z Gathering data for ns/openshift-image-registry...
2026-07-01T17:25:33.747103064Z Gathering data for ns/openshift-ingress-operator...
2026-07-01T17:25:33.826502380Z Done executing Istio gather script
2026-07-01T17:25:33.850899955Z Waiting for performance profile collector pods to become ready: 2
2026-07-01T17:25:34.109815478Z Gathering data for ns/openshift-ingress...
2026-07-01T17:25:34.465841444Z Gathering data for ns/openshift-ingress-canary...
2026-07-01T17:25:34.926488470Z Gathering data for ns/openshift-insights...
2026-07-01T17:25:34.977466647Z Waiting for performance profile collector pods to become ready: 3
2026-07-01T17:25:36.095078887Z Waiting for performance profile collector pods to become ready: 4
2026-07-01T17:25:36.440503300Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-01T17:25:37.218197487Z Waiting for performance profile collector pods to become ready: 5
2026-07-01T17:25:37.333890251Z Gathering data for ns/kserve...
2026-07-01T17:25:38.334803200Z Waiting for performance profile collector pods to become ready: 6
2026-07-01T17:25:39.440166067Z Gathering data for ns/openshift-config...
2026-07-01T17:25:39.450658259Z Waiting for performance profile collector pods to become ready: 7
2026-07-01T17:25:39.773709147Z Gathering data for ns/openshift-config-managed...
2026-07-01T17:25:40.215182246Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-01T17:25:40.538135090Z Gathering data for ns/openshift-kube-apiserver...
2026-07-01T17:25:40.589903421Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-01T17:25:40.852917917Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-01T17:25:40.877487010Z Collecting performance related data for node ip-10-0-139-133.ec2.internal
2026-07-01T17:25:40.886616768Z Collecting performance related data for node ip-10-0-137-244.ec2.internal
2026-07-01T17:25:40.897352622Z Collecting performance related data for node ip-10-0-138-194.ec2.internal
2026-07-01T17:25:41.228215811Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-01T17:25:41.446542522Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-01T17:25:41.642026982Z Gathering data for ns/openshift-kube-scheduler...
2026-07-01T17:25:42.014186114Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-01T17:25:42.382326148Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-01T17:25:42.767671041Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-01T17:25:43.110323538Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-01T17:25:43.704124648Z Gathering data for ns/openshift-multus...
2026-07-01T17:25:45.546261519Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-01T17:25:46.451162128Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-01T17:25:47.181578027Z Gathering data for ns/openshift-host-network...
2026-07-01T17:25:47.543721507Z Gathering data for ns/openshift-network-diagnostics...
2026-07-01T17:25:48.175073663Z Gathering data for ns/openshift-network-node-identity...
2026-07-01T17:25:48.707214220Z Gathering data for ns/openshift-network-operator...
2026-07-01T17:25:49.150506588Z Collecting kubelet logs for node ip-10-0-139-133.ec2.internal
2026-07-01T17:25:49.152279176Z Collecting kubelet logs for node ip-10-0-137-244.ec2.internal
2026-07-01T17:25:49.153916648Z Collecting kubelet logs for node ip-10-0-138-194.ec2.internal
2026-07-01T17:25:49.451662568Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-01T17:25:49.560496067Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-4lxfx namespace
2026-07-01T17:25:49.568883100Z INFO: Node performance data collection complete.
2026-07-01T17:25:49.860444574Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-01T17:25:50.433128484Z Gathering data for ns/openshift-apiserver-operator...
2026-07-01T17:25:50.770401415Z Gathering data for ns/openshift-apiserver...
2026-07-01T17:25:51.201959784Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-01T17:25:51.456112510Z [disk usage checker] Volume usage percentage: current = 12 ; allowed = 70
2026-07-01T17:25:51.483764318Z Gathering data for ns/openshift-controller-manager...
2026-07-01T17:25:51.801878472Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-01T17:25:52.496077159Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-01T17:25:52.816008532Z Gathering data for ns/openshift-service-ca-operator...
2026-07-01T17:25:53.159382215Z Gathering data for ns/openshift-service-ca...
2026-07-01T17:25:53.543700802Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-01T17:25:54.581824998Z Wrote inspect data to must-gather.
2026-07-01T17:25:54.581868420Z error: inspection completed with the errors occurred while gathering data:
2026-07-01T17:25:54.581868420Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-01T17:25:54.717289676Z error: the server doesn't have a resource type "clusters"
2026-07-01T17:25:54.864631582Z Caches written to disk