[must-gather ] OUT 2026-07-31T17:01:32.97954156Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: f5ddae8d-8631-48a7-9732-b9db86c0e9f6 ClientVersion: 4.22.5 ClusterVersion: Stable at "4.21.27" ClusterOperators: clusteroperator/authentication is missing clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-07-31T17:01:33.041294729Z namespace/openshift-must-gather-xpswm created [must-gather ] OUT 2026-07-31T17:01:33.049729514Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-2t5b4 created [must-gather ] OUT 2026-07-31T17:01:33.070352275Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-sgdd9] OUT 2026-07-31T17:02:27.133155171Z waiting for gather to complete [must-gather-sgdd9] OUT 2026-07-31T17:02:27.136460259Z downloading gather output [must-gather-sgdd9] OUT 2026-07-31T17:02:27.521148847Z receiving incremental file list [must-gather-sgdd9] OUT 2026-07-31T17:02:27.536547125Z ./ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.536632207Z aggregated-discovery-api.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.566560439Z aggregated-discovery-apis.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.56773253Z event-filter.html [must-gather-sgdd9] OUT 2026-07-31T17:02:27.567843753Z timestamp [must-gather-sgdd9] OUT 2026-07-31T17:02:27.567954956Z version [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568069889Z namespaces/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568081539Z namespaces/opendatahub/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568131521Z namespaces/opendatahub/opendatahub.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568216113Z namespaces/opendatahub/apps.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568249404Z namespaces/opendatahub/apps.openshift.io/deploymentconfigs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568289875Z namespaces/opendatahub/apps/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568338076Z namespaces/opendatahub/apps/daemonsets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568433169Z namespaces/opendatahub/apps/deployments.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568546582Z namespaces/opendatahub/apps/replicasets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568659015Z namespaces/opendatahub/apps/statefulsets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568695546Z namespaces/opendatahub/autoscaling/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568743977Z namespaces/opendatahub/autoscaling/horizontalpodautoscalers.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568792288Z namespaces/opendatahub/batch/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568829429Z namespaces/opendatahub/batch/cronjobs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568912141Z namespaces/opendatahub/batch/jobs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.568960533Z namespaces/opendatahub/build.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569001584Z namespaces/opendatahub/build.openshift.io/buildconfigs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569083366Z namespaces/opendatahub/build.openshift.io/builds.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569152468Z namespaces/opendatahub/core/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569191819Z namespaces/opendatahub/core/configmaps.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569319282Z namespaces/opendatahub/core/endpoints.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569404234Z namespaces/opendatahub/core/events.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569524518Z namespaces/opendatahub/core/persistentvolumeclaims.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.5696064Z namespaces/opendatahub/core/pods.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569742323Z namespaces/opendatahub/core/replicationcontrollers.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.569864447Z namespaces/opendatahub/core/secrets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570049531Z namespaces/opendatahub/core/services.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570106233Z namespaces/opendatahub/discovery.k8s.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570170095Z namespaces/opendatahub/discovery.k8s.io/endpointslices.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570233536Z namespaces/opendatahub/image.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570280858Z namespaces/opendatahub/image.openshift.io/imagestreams.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.57037911Z namespaces/opendatahub/k8s.ovn.org/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570398151Z namespaces/opendatahub/k8s.ovn.org/egressfirewalls.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570497263Z namespaces/opendatahub/k8s.ovn.org/egressqoses.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570533564Z namespaces/opendatahub/monitoring.coreos.com/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570572235Z namespaces/opendatahub/monitoring.coreos.com/servicemonitors.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570616416Z namespaces/opendatahub/networking.k8s.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570654088Z namespaces/opendatahub/networking.k8s.io/networkpolicies.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570692089Z namespaces/opendatahub/pods/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570701319Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.57074175Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/kubeflow-trainer-controller-manager-99b9897b5-9nhmv.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570830572Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/manager/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570839672Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/manager/manager/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570844813Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/manager/manager/logs/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.570871263Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/manager/manager/logs/current.log [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572424964Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/manager/manager/logs/previous.insecure.log [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572492186Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-99b9897b5-9nhmv/manager/manager/logs/previous.log [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572514267Z namespaces/opendatahub/policy/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572565038Z namespaces/opendatahub/policy/poddisruptionbudgets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.5726418Z namespaces/opendatahub/route.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572686331Z namespaces/opendatahub/route.openshift.io/routes.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572732903Z namespaces/openshift/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572780224Z namespaces/openshift/openshift.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572887876Z namespaces/openshift/apps.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572906577Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.572966779Z namespaces/openshift/apps/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.57301202Z namespaces/openshift/apps/daemonsets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573135813Z namespaces/openshift/apps/deployments.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573237626Z namespaces/openshift/apps/replicasets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573340088Z namespaces/openshift/apps/statefulsets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.57338774Z namespaces/openshift/autoscaling/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573449842Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573509153Z namespaces/openshift/batch/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573552194Z namespaces/openshift/batch/cronjobs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573660717Z namespaces/openshift/batch/jobs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573720469Z namespaces/openshift/build.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.57376308Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573867893Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573921094Z namespaces/openshift/core/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.573979165Z namespaces/openshift/core/configmaps.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574110319Z namespaces/openshift/core/endpoints.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574213742Z namespaces/openshift/core/events.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574305174Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574401807Z namespaces/openshift/core/pods.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574495369Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574636913Z namespaces/openshift/core/secrets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574820848Z namespaces/openshift/core/services.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574884559Z namespaces/openshift/discovery.k8s.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.574940791Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.575000602Z namespaces/openshift/image.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.575050674Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.576660736Z namespaces/openshift/k8s.ovn.org/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.576708318Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.57681026Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.576868792Z namespaces/openshift/monitoring.coreos.com/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.576929223Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.577001575Z namespaces/openshift/networking.k8s.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.577051467Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.577132659Z namespaces/openshift/policy/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.5771851Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.577248572Z namespaces/openshift/route.openshift.io/ [must-gather-sgdd9] OUT 2026-07-31T17:02:27.577296003Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-sgdd9] OUT 2026-07-31T17:02:27.589303241Z [must-gather-sgdd9] OUT 2026-07-31T17:02:27.589341012Z sent 1,380 bytes received 98,895 bytes 200,550.00 bytes/sec [must-gather-sgdd9] OUT 2026-07-31T17:02:27.589345512Z total size is 1,269,696 speedup is 12.66 [must-gather ] OUT 2026-07-31T17:02:27.620565198Z namespace/openshift-must-gather-xpswm deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: f5ddae8d-8631-48a7-9732-b9db86c0e9f6 ClientVersion: 4.22.5 ClusterVersion: Stable at "4.21.27" ClusterOperators: clusteroperator/authentication is missing clusteroperator/cloud-credential is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing