2026-07-28T18:06:25.335028866Z [disk usage checker] Started
2026-07-28T18:06:25.338465787Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T18:06:25.534281830Z Gathering data for ns/openshift-cluster-version...
2026-07-28T18:06:25.694604324Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T18:06:25.799489474Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T18:06:25.893050911Z Gathering data for ns/default...
2026-07-28T18:06:26.260867214Z Gathering data for ns/openshift...
2026-07-28T18:06:26.746156708Z Gathering data for ns/kube-system...
2026-07-28T18:06:27.620315122Z Gathering data for ns/openshift-etcd...
2026-07-28T18:06:28.000658588Z Wrote inspect data to must-gather.
2026-07-28T18:06:28.000695312Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T18:06:28.000695312Z     namespaces "assisted-installer" not found
2026-07-28T18:06:28.176625650Z Waiting on subprocesses to finish execution.
2026-07-28T18:06:28.189702245Z Executing Istio gather script
2026-07-28T18:06:28.190530850Z INFO: Gathering machine config daemon's old logs from all nodes
2026-07-28T18:06:28.190702540Z INFO: Gathering HAProxy config files
2026-07-28T18:06:28.191163142Z INFO: Gathering on-disk MachineConfig from degraded nodes
2026-07-28T18:06:28.192900253Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time.
2026-07-28T18:06:28.225265295Z INFO: Collecting host service logs for crio
2026-07-28T18:06:28.225643739Z INFO: Collecting host service logs for kubelet
2026-07-28T18:06:28.225950094Z INFO: Collecting host service logs for rpm-ostreed
2026-07-28T18:06:28.226256116Z INFO: Collecting host service logs for ostree-finalize-staged
2026-07-28T18:06:28.226667705Z INFO: Collecting host service logs for machine-config-daemon-firstboot
2026-07-28T18:06:28.227024905Z INFO: Collecting host service logs for machine-config-daemon-host
2026-07-28T18:06:28.227344800Z INFO: Collecting host service logs for NetworkManager
2026-07-28T18:06:28.227679257Z INFO: Collecting host service logs for openvswitch
2026-07-28T18:06:28.227988029Z INFO: Collecting host service logs for ovs-configuration
2026-07-28T18:06:28.228304481Z INFO: Collecting host service logs for ovsdb-server
2026-07-28T18:06:28.228669634Z INFO: Collecting host service logs for ovs-vswitchd
2026-07-28T18:06:28.229041474Z INFO: Waiting for worker host service log collection to complete ...
2026-07-28T18:06:28.566840889Z INFO: Collecting Insights Archives from insights-operator-5885fdb759-fdb6z
2026-07-28T18:06:28.566936936Z insights-runtime-extractor-9cghg
2026-07-28T18:06:28.566957618Z insights-runtime-extractor-f7nmc
2026-07-28T18:06:28.566976058Z insights-runtime-extractor-rflm7
2026-07-28T18:06:28.729217783Z INFO: Waiting for node performance related collection to complete ...
2026-07-28T18:06:28.850129635Z INFO: "kubernetes-nmstate-operator" not detected. Skipping.
2026-07-28T18:06:29.072656109Z INFO: "metallb-operator" not detected. Skipping.
2026-07-28T18:06:29.174676263Z INFO: "sriov-network-operator" not detected. Skipping.
2026-07-28T18:06:29.259394623Z error: the server doesn't have a resource type "performanceprofile"
2026-07-28T18:06:29.284500989Z INFO: Found 1 replicas - prometheus-k8s-0
2026-07-28T18:06:29.301894542Z No resources found
2026-07-28T18:06:29.604467727Z ERROR: No running kube-apiserver pods found
2026-07-28T18:06:29.619503594Z No resources found in openshift-etcd namespace.
2026-07-28T18:06:29.696373156Z INFO: namespace openshift-frr-k8s not detected. Skipping.
2026-07-28T18:06:29.812103737Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments
2026-07-28T18:06:29.812103737Z See 'oc rsync -h' for help and examples
2026-07-28T18:06:29.878239026Z Wrote inspect data to must-gather.
2026-07-28T18:06:29.884783582Z INFO: Waiting for on-disk MachineConfig collection to complete ...
2026-07-28T18:06:29.884856568Z INFO: on-disk MachineConfig config collection complete.
2026-07-28T18:06:30.037410633Z INFO: Worker host service log collection to complete.
2026-07-28T18:06:30.169333138Z Wrote inspect data to must-gather.
2026-07-28T18:06:30.312291191Z INFO: Waiting for HAProxy config collection to complete ...
2026-07-28T18:06:30.356789160Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T18:06:30.405606870Z Wrote inspect data to must-gather.
2026-07-28T18:06:30.462079701Z INFO: Getting alertmanagers from prometheus-k8s-0
2026-07-28T18:06:30.606839824Z Wrote inspect data to must-gather.
2026-07-28T18:06:30.640212559Z INFO: 'previous-logs' folder not found on ip-10-0-129-206.ec2.internal, skipping...
2026-07-28T18:06:30.661069191Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T18:06:30.748627998Z Wrote inspect data to must-gather.
2026-07-28T18:06:30.770429489Z error: the server doesn't have a resource type "multi-networkpolicy"
2026-07-28T18:06:30.843946301Z Gathering data for ns/openshift-monitoring...
2026-07-28T18:06:30.860095317Z No resources found
2026-07-28T18:06:30.870445817Z error: the server doesn't have a resource type "machineconfigs"
2026-07-28T18:06:30.907592394Z Wrote inspect data to must-gather.
2026-07-28T18:06:30.994100205Z Wrote inspect data to must-gather.
2026-07-28T18:06:31.073718027Z tar: Removing leading `/' from member names
2026-07-28T18:06:31.080896846Z INFO: HAProxy config collection complete.
2026-07-28T18:06:31.136704179Z error: the server doesn't have a resource type "machineconfigpools"
2026-07-28T18:06:31.153076156Z INFO: 'previous-logs' folder not found on ip-10-0-134-28.ec2.internal, skipping...
2026-07-28T18:06:31.251171784Z INFO: Getting rules from prometheus-k8s-0
2026-07-28T18:06:31.295097760Z Wrote inspect data to must-gather.
2026-07-28T18:06:31.425943844Z Wrote inspect data to must-gather.
2026-07-28T18:06:31.502367766Z INFO: 'previous-logs' folder not found on ip-10-0-136-215.ec2.internal, skipping...
2026-07-28T18:06:31.503771567Z INFO: Waiting for Machine Config Daemon termination log collection to complete ...
2026-07-28T18:06:31.503849954Z INFO: Machine Config Daemon termination log collection complete.
2026-07-28T18:06:31.615677594Z error: the server doesn't have a resource type "kubeletconfigs"
2026-07-28T18:06:31.619353546Z Wrote inspect data to must-gather.
2026-07-28T18:06:31.722013407Z INFO: Getting status/config from prometheus-k8s-0
2026-07-28T18:06:31.880510260Z Wrote inspect data to must-gather.
2026-07-28T18:06:31.893005270Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection.
2026-07-28T18:06:31.930824731Z Wrote inspect data to must-gather.
2026-07-28T18:06:32.122357915Z Wrote inspect data to must-gather.
2026-07-28T18:06:32.148030440Z INFO: Getting status/flags from prometheus-k8s-0
2026-07-28T18:06:32.257392190Z INFO: INTERCONNECT MODE
2026-07-28T18:06:32.257430914Z INFO: Gathering ovn-kubernetes DBs
2026-07-28T18:06:32.340302105Z Wrote inspect data to must-gather.
2026-07-28T18:06:32.389639990Z INFO: Getting status/runtimeinfo from prometheus-k8s-0
2026-07-28T18:06:32.440033532Z INFO: Gathering OVN_Northbound from ovnkube-node-c29m8...
2026-07-28T18:06:32.440294209Z INFO: Gathering OVN_Northbound from ovnkube-node-fjw6m...
2026-07-28T18:06:32.440618360Z INFO: Gathering OVN_Northbound from ovnkube-node-vcdd5...
2026-07-28T18:06:32.440928361Z INFO: Gathering OVN_Southbound from ovnkube-node-c29m8...
2026-07-28T18:06:32.441203574Z INFO: Gathering OVN_Southbound from ovnkube-node-fjw6m...
2026-07-28T18:06:32.441492265Z INFO: Gathering OVN_Southbound from ovnkube-node-vcdd5...
2026-07-28T18:06:32.648745924Z Wrote inspect data to must-gather.
2026-07-28T18:06:32.727399055Z INFO: Getting targets?state=active from prometheus-k8s-0
2026-07-28T18:06:32.802934262Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io
2026-07-28T18:06:32.810401726Z tar: Removing leading `/' from member names
2026-07-28T18:06:32.821094933Z tar: Removing leading `/' from member names
2026-07-28T18:06:32.825786904Z tar: Removing leading `/' from member names
2026-07-28T18:06:32.827229235Z tar: Removing leading `/' from member names
2026-07-28T18:06:32.856843264Z tar: Removing leading `/' from member names
2026-07-28T18:06:32.865824498Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2026-07-28T18:06:32.894271242Z tar: Removing leading `/' from member names
2026-07-28T18:06:32.966058704Z Wrote inspect data to must-gather.
2026-07-28T18:06:33.039626639Z INFO: Getting status/tsdb from prometheus-k8s-0
2026-07-28T18:06:33.045508125Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:33.245639585Z Wrote inspect data to must-gather.
2026-07-28T18:06:33.309692125Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:33.313009167Z Inspecting resource crd/gateways.gateway.networking.k8s.io
2026-07-28T18:06:33.431295756Z INFO: Waiting for network log collection to complete ...
2026-07-28T18:06:33.431336791Z INFO: Waiting for ovnk database copies to complete ...
2026-07-28T18:06:33.431747386Z INFO: Copying ovnk databases complete.
2026-07-28T18:06:33.433310410Z 3.8M	must-gather/network_logs/ovnk_database_store
2026-07-28T18:06:33.436006429Z ovnk_database_store/
2026-07-28T18:06:33.436006429Z ovnk_database_store/ovnkube-node-c29m8_sbdb
2026-07-28T18:06:33.465973956Z ovnk_database_store/ovnkube-node-c29m8_nbdb
2026-07-28T18:06:33.474185939Z W0728 18:06:33.474143    1507 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in ''
2026-07-28T18:06:33.475565688Z ovnk_database_store/ovnkube-node-fjw6m_nbdb
2026-07-28T18:06:33.483848373Z Wrote inspect data to must-gather.
2026-07-28T18:06:33.486138148Z ovnk_database_store/ovnkube-node-vcdd5_sbdb
2026-07-28T18:06:33.510597832Z ovnk_database_store/ovnkube-node-fjw6m_sbdb
2026-07-28T18:06:33.523502025Z INFO: Getting status from alertmanager-main-0
2026-07-28T18:06:33.536968921Z ovnk_database_store/ovnkube-node-vcdd5_nbdb
2026-07-28T18:06:33.552875421Z INFO: Network log collection complete.
2026-07-28T18:06:33.561473625Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
2026-07-28T18:06:33.580392555Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:33.776066294Z Gathering data for ns/openshift-network-console...
2026-07-28T18:06:33.793641334Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found
2026-07-28T18:06:33.795528781Z INFO: Fallback to identify the container image from release info
2026-07-28T18:06:33.812670183Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:33.815084725Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io
2026-07-28T18:06:34.024960947Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:34.247700228Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:34.250428004Z Inspecting resource crd/httproutes.gateway.networking.k8s.io
2026-07-28T18:06:34.297527875Z Gathering data for ns/openshift-console-operator...
2026-07-28T18:06:34.469995876Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:34.684569888Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:34.687217467Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811
2026-07-28T18:06:34.688262414Z Inspecting resource crd/inferencepools.inference.networking.k8s.io
2026-07-28T18:06:34.754612187Z Gathering data for ns/openshift-console...
2026-07-28T18:06:34.827828301Z daemonset.apps/perf-node-gather-daemonset created
2026-07-28T18:06:34.872775803Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:34.988239254Z Waiting for performance profile collector pods to become ready: 1
2026-07-28T18:06:35.076413403Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:35.079281309Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io
2026-07-28T18:06:35.273025343Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:35.368077936Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T18:06:35.425642167Z Gathering data for ns/openshift-cluster-storage-operator...
2026-07-28T18:06:35.504931935Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:35.507970104Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io
2026-07-28T18:06:35.680929637Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:35.888878667Z Wrote inspect data to /must-gather/istio.
2026-07-28T18:06:36.022127414Z Gathering data for ns/openshift-dns-operator...
2026-07-28T18:06:36.149045948Z Waiting for performance profile collector pods to become ready: 2
2026-07-28T18:06:36.398911833Z Gathering data for ns/openshift-dns...
2026-07-28T18:06:36.611305062Z Done executing Istio gather script
2026-07-28T18:06:37.125729056Z Gathering data for ns/openshift-image-registry...
2026-07-28T18:06:37.261528003Z Waiting for performance profile collector pods to become ready: 3
2026-07-28T18:06:37.637735590Z Gathering data for ns/openshift-ingress-operator...
2026-07-28T18:06:38.079052594Z Gathering data for ns/openshift-ingress...
2026-07-28T18:06:38.392452609Z Waiting for performance profile collector pods to become ready: 4
2026-07-28T18:06:38.466584732Z Gathering data for ns/openshift-ingress-canary...
2026-07-28T18:06:38.955762124Z Gathering data for ns/openshift-insights...
2026-07-28T18:06:39.505538149Z Waiting for performance profile collector pods to become ready: 5
2026-07-28T18:06:40.372818807Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T18:06:40.625210998Z Waiting for performance profile collector pods to become ready: 6
2026-07-28T18:06:41.532856571Z Gathering data for ns/kserve...
2026-07-28T18:06:41.747509665Z Waiting for performance profile collector pods to become ready: 7
2026-07-28T18:06:42.877325829Z Daemonset perf-node-gather-daemonset ready 3 out of 3
2026-07-28T18:06:43.151717273Z Collecting performance related data for node ip-10-0-129-206.ec2.internal
2026-07-28T18:06:43.159430198Z Collecting performance related data for node ip-10-0-134-28.ec2.internal
2026-07-28T18:06:43.166457080Z Collecting performance related data for node ip-10-0-136-215.ec2.internal
2026-07-28T18:06:43.291939565Z Gathering data for ns/openshift-config...
2026-07-28T18:06:43.640139425Z Gathering data for ns/openshift-config-managed...
2026-07-28T18:06:44.182302519Z Gathering data for ns/openshift-kube-apiserver-operator...
2026-07-28T18:06:44.584146846Z Gathering data for ns/openshift-kube-apiserver...
2026-07-28T18:06:44.967604271Z Gathering data for ns/openshift-kube-controller-manager...
2026-07-28T18:06:45.350803784Z Gathering data for ns/openshift-kube-controller-manager-operator...
2026-07-28T18:06:45.380516557Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T18:06:45.753993655Z Gathering data for ns/openshift-kube-scheduler...
2026-07-28T18:06:46.119165271Z Gathering data for ns/openshift-kube-scheduler-operator...
2026-07-28T18:06:46.502368823Z Gathering data for ns/openshift-kube-storage-version-migrator...
2026-07-28T18:06:46.959183931Z Gathering data for ns/openshift-kube-storage-version-migrator-operator...
2026-07-28T18:06:47.445697068Z Gathering data for ns/openshift-user-workload-monitoring...
2026-07-28T18:06:48.116121976Z Gathering data for ns/openshift-multus...
2026-07-28T18:06:49.854277804Z Gathering data for ns/openshift-ovn-kubernetes...
2026-07-28T18:06:50.387193525Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T18:06:51.351342004Z Gathering data for ns/openshift-host-network...
2026-07-28T18:06:51.641635454Z Collecting kubelet logs for node ip-10-0-129-206.ec2.internal
2026-07-28T18:06:51.644130209Z Collecting kubelet logs for node ip-10-0-134-28.ec2.internal
2026-07-28T18:06:51.646332444Z Collecting kubelet logs for node ip-10-0-136-215.ec2.internal
2026-07-28T18:06:51.777136284Z Gathering data for ns/openshift-network-diagnostics...
2026-07-28T18:06:52.027936312Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-92xdj namespace
2026-07-28T18:06:52.036167155Z INFO: Node performance data collection complete.
2026-07-28T18:06:52.437327020Z Gathering data for ns/openshift-network-node-identity...
2026-07-28T18:06:52.958290372Z Gathering data for ns/openshift-network-operator...
2026-07-28T18:06:53.443867776Z Gathering data for ns/openshift-cloud-network-config-controller...
2026-07-28T18:06:53.736315845Z Gathering data for ns/openshift-cluster-node-tuning-operator...
2026-07-28T18:06:54.313876174Z Gathering data for ns/openshift-apiserver-operator...
2026-07-28T18:06:54.673880551Z Gathering data for ns/openshift-apiserver...
2026-07-28T18:06:55.154385711Z Gathering data for ns/openshift-controller-manager-operator...
2026-07-28T18:06:55.391722156Z [disk usage checker] Volume usage percentage: current = 13 ; allowed = 70
2026-07-28T18:06:55.537708051Z Gathering data for ns/openshift-controller-manager...
2026-07-28T18:06:55.913593492Z Gathering data for ns/openshift-cluster-samples-operator...
2026-07-28T18:06:56.642607892Z Gathering data for ns/openshift-operator-lifecycle-manager...
2026-07-28T18:06:56.990561412Z Gathering data for ns/openshift-service-ca-operator...
2026-07-28T18:06:57.383220427Z Gathering data for ns/openshift-service-ca...
2026-07-28T18:06:57.793149085Z Gathering data for ns/openshift-cluster-csi-drivers...
2026-07-28T18:06:58.845988987Z Wrote inspect data to must-gather.
2026-07-28T18:06:58.846031851Z error: inspection completed with the errors occurred while gathering data:
2026-07-28T18:06:58.846031851Z     skipping gathering secrets/support due to error: secrets "support" not found
2026-07-28T18:06:58.976703070Z error: the server doesn't have a resource type "clusters"
2026-07-28T18:06:59.131557812Z Caches written to disk