[must-gather ] OUT 2026-10-31T00:11:44.082739917Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: d219e71a-432e-404c-98c8-577766f251b5 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-10-31T00:11:44.203269989Z namespace/openshift-must-gather-9zkzh created [must-gather ] OUT 2026-10-31T00:11:44.218523866Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-sl4m7 created [must-gather ] OUT 2026-10-31T00:11:44.257942122Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-lbjrc] OUT 2026-10-31T00:15:07.492494602Z waiting for gather to complete [must-gather-lbjrc] OUT 2026-10-31T00:15:07.507374818Z downloading gather output [must-gather-lbjrc] OUT 2026-10-31T00:15:08.520285148Z receiving incremental file list [must-gather-lbjrc] OUT 2026-10-31T00:15:08.647532357Z ./ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.647596108Z aggregated-discovery-api.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.648001576Z aggregated-discovery-apis.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.648817762Z event-filter.html [must-gather-lbjrc] OUT 2026-10-31T00:15:08.648988945Z timestamp [must-gather-lbjrc] OUT 2026-10-31T00:15:08.658209602Z version [must-gather-lbjrc] OUT 2026-10-31T00:15:08.696818285Z namespaces/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.696858106Z namespaces/opendatahub/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697014159Z namespaces/opendatahub/opendatahub.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697159172Z namespaces/opendatahub/apps.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697229593Z namespaces/opendatahub/apps.openshift.io/deploymentconfigs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697302735Z namespaces/opendatahub/apps/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697387776Z namespaces/opendatahub/apps/daemonsets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697505038Z namespaces/opendatahub/apps/deployments.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697668772Z namespaces/opendatahub/apps/replicasets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697809404Z namespaces/opendatahub/apps/statefulsets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697909016Z namespaces/opendatahub/autoscaling/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.697988018Z namespaces/opendatahub/autoscaling/horizontalpodautoscalers.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698052199Z namespaces/opendatahub/batch/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.69812295Z namespaces/opendatahub/batch/cronjobs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698250153Z namespaces/opendatahub/batch/jobs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698313184Z namespaces/opendatahub/build.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698387595Z namespaces/opendatahub/build.openshift.io/buildconfigs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698502318Z namespaces/opendatahub/build.openshift.io/builds.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698569199Z namespaces/opendatahub/core/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.69863096Z namespaces/opendatahub/core/configmaps.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698815794Z namespaces/opendatahub/core/endpoints.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.698961586Z namespaces/opendatahub/core/events.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.69912692Z namespaces/opendatahub/core/persistentvolumeclaims.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.699241222Z namespaces/opendatahub/core/pods.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.699395335Z namespaces/opendatahub/core/replicationcontrollers.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.699605539Z namespaces/opendatahub/core/secrets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.699815293Z namespaces/opendatahub/core/services.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.699929495Z namespaces/opendatahub/discovery.k8s.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.699993957Z namespaces/opendatahub/discovery.k8s.io/endpointslices.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.700079158Z namespaces/opendatahub/image.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.700143639Z namespaces/opendatahub/image.openshift.io/imagestreams.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.707530291Z namespaces/opendatahub/k8s.ovn.org/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.707625433Z namespaces/opendatahub/k8s.ovn.org/egressfirewalls.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.707770886Z namespaces/opendatahub/k8s.ovn.org/egressqoses.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.707835357Z namespaces/opendatahub/monitoring.coreos.com/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.7079647Z namespaces/opendatahub/monitoring.coreos.com/servicemonitors.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708025441Z namespaces/opendatahub/networking.k8s.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708103913Z namespaces/opendatahub/networking.k8s.io/networkpolicies.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708170694Z namespaces/opendatahub/pods/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708180974Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708238355Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708349057Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/manager/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708361638Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/manager/manager/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708368248Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/manager/manager/logs/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.708413368Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/manager/manager/logs/current.log [must-gather-lbjrc] OUT 2026-10-31T00:15:08.709750374Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/manager/manager/logs/previous.insecure.log [must-gather-lbjrc] OUT 2026-10-31T00:15:08.709893497Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-bgzjw/manager/manager/logs/previous.log [must-gather-lbjrc] OUT 2026-10-31T00:15:08.709966398Z namespaces/opendatahub/policy/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710034119Z namespaces/opendatahub/policy/poddisruptionbudgets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710098871Z namespaces/opendatahub/route.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710153672Z namespaces/opendatahub/route.openshift.io/routes.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710221243Z namespaces/openshift/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710279074Z namespaces/openshift/openshift.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710347426Z namespaces/openshift/apps.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710393777Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710451388Z namespaces/openshift/apps/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710515199Z namespaces/openshift/apps/daemonsets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710650102Z namespaces/openshift/apps/deployments.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710749813Z namespaces/openshift/apps/replicasets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710897176Z namespaces/openshift/apps/statefulsets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.710978168Z namespaces/openshift/autoscaling/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711042479Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.7111036Z namespaces/openshift/batch/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711158711Z namespaces/openshift/batch/cronjobs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711266193Z namespaces/openshift/batch/jobs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711331595Z namespaces/openshift/build.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711396766Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711500708Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711559449Z namespaces/openshift/core/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.71162169Z namespaces/openshift/core/configmaps.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711768353Z namespaces/openshift/core/endpoints.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.711859025Z namespaces/openshift/core/events.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.712011868Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.71211781Z namespaces/openshift/core/pods.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.712232332Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.712400485Z namespaces/openshift/core/secrets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.712592189Z namespaces/openshift/core/services.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.71265624Z namespaces/openshift/discovery.k8s.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.712711131Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.712770562Z namespaces/openshift/image.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.712821093Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.714613858Z namespaces/openshift/k8s.ovn.org/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.714665599Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.714789311Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.714814612Z namespaces/openshift/monitoring.coreos.com/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.714899993Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.715016295Z namespaces/openshift/networking.k8s.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.715085657Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.715151088Z namespaces/openshift/policy/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.715208089Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.71527002Z namespaces/openshift/route.openshift.io/ [must-gather-lbjrc] OUT 2026-10-31T00:15:08.715321531Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-lbjrc] OUT 2026-10-31T00:15:08.838284517Z [must-gather-lbjrc] OUT 2026-10-31T00:15:08.838321548Z sent 1,384 bytes received 98,528 bytes 66,608.00 bytes/sec [must-gather-lbjrc] OUT 2026-10-31T00:15:08.838329188Z total size is 1,194,760 speedup is 11.96 [must-gather ] OUT 2026-10-31T00:15:08.939303171Z namespace/openshift-must-gather-9zkzh deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: d219e71a-432e-404c-98c8-577766f251b5 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing