[must-gather ] OUT 2026-10-30T11:54:34.087459273Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 3a327ebf-8bb6-4092-bbf5-2613aef6a4e3 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-10-30T11:54:34.201565007Z namespace/openshift-must-gather-26zjn created [must-gather ] OUT 2026-10-30T11:54:34.21811985Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-xk7d5 created [must-gather ] OUT 2026-10-30T11:54:34.257208465Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-dsz7b] OUT 2026-10-30T11:56:04.132185179Z waiting for gather to complete [must-gather-dsz7b] OUT 2026-10-30T11:56:04.144929879Z downloading gather output [must-gather-dsz7b] OUT 2026-10-30T11:56:04.641744727Z receiving incremental file list [must-gather-dsz7b] OUT 2026-10-30T11:56:04.697581829Z ./ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.697688532Z aggregated-discovery-api.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.698093422Z aggregated-discovery-apis.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.711372566Z event-filter.html [must-gather-dsz7b] OUT 2026-10-30T11:56:04.71152709Z timestamp [must-gather-dsz7b] OUT 2026-10-30T11:56:04.711658383Z version [must-gather-dsz7b] OUT 2026-10-30T11:56:04.720381662Z namespaces/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.720411702Z namespaces/opendatahub/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.720490924Z namespaces/opendatahub/opendatahub.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.720614008Z namespaces/opendatahub/apps.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.72068638Z namespaces/opendatahub/apps.openshift.io/deploymentconfigs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.720757181Z namespaces/opendatahub/apps/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.720838363Z namespaces/opendatahub/apps/daemonsets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.720957136Z namespaces/opendatahub/apps/deployments.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.72110932Z namespaces/opendatahub/apps/replicasets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721251554Z namespaces/opendatahub/apps/statefulsets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721299025Z namespaces/opendatahub/autoscaling/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721381557Z namespaces/opendatahub/autoscaling/horizontalpodautoscalers.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721455779Z namespaces/opendatahub/batch/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.7214904Z namespaces/opendatahub/batch/cronjobs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721594902Z namespaces/opendatahub/batch/jobs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721649064Z namespaces/opendatahub/build.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721706985Z namespaces/opendatahub/build.openshift.io/buildconfigs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721818808Z namespaces/opendatahub/build.openshift.io/builds.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721864769Z namespaces/opendatahub/core/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.721927731Z namespaces/opendatahub/core/configmaps.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.722108425Z namespaces/opendatahub/core/endpoints.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.722207098Z namespaces/opendatahub/core/events.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.722390712Z namespaces/opendatahub/core/persistentvolumeclaims.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.722482555Z namespaces/opendatahub/core/pods.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.722656009Z namespaces/opendatahub/core/replicationcontrollers.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.722797202Z namespaces/opendatahub/core/secrets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.723001268Z namespaces/opendatahub/core/services.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.72309656Z namespaces/opendatahub/discovery.k8s.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.723188872Z namespaces/opendatahub/discovery.k8s.io/endpointslices.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.723302235Z namespaces/opendatahub/image.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.723404088Z namespaces/opendatahub/image.openshift.io/imagestreams.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.723576902Z namespaces/opendatahub/k8s.ovn.org/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.723660274Z namespaces/opendatahub/k8s.ovn.org/egressfirewalls.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.733562213Z namespaces/opendatahub/k8s.ovn.org/egressqoses.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.733626705Z namespaces/opendatahub/monitoring.coreos.com/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.733699896Z namespaces/opendatahub/monitoring.coreos.com/servicemonitors.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.733778868Z namespaces/opendatahub/networking.k8s.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.73384852Z namespaces/opendatahub/networking.k8s.io/networkpolicies.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.733925512Z namespaces/opendatahub/pods/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.733933912Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.733994594Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.734108237Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/manager/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.734117557Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/manager/manager/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.734122447Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/manager/manager/logs/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.734173888Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/manager/manager/logs/current.log [must-gather-dsz7b] OUT 2026-10-30T11:56:04.747609366Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/manager/manager/logs/previous.insecure.log [must-gather-dsz7b] OUT 2026-10-30T11:56:04.747757939Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-fgw9c/manager/manager/logs/previous.log [must-gather-dsz7b] OUT 2026-10-30T11:56:04.747831071Z namespaces/opendatahub/policy/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.747926224Z namespaces/opendatahub/policy/poddisruptionbudgets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748004035Z namespaces/opendatahub/route.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748071677Z namespaces/opendatahub/route.openshift.io/routes.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748157869Z namespaces/openshift/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748231161Z namespaces/openshift/openshift.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748368725Z namespaces/openshift/apps.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748446827Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748528039Z namespaces/openshift/apps/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.74859514Z namespaces/openshift/apps/daemonsets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748727534Z namespaces/openshift/apps/deployments.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.748861917Z namespaces/openshift/apps/replicasets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.74899762Z namespaces/openshift/apps/statefulsets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.749425691Z namespaces/openshift/autoscaling/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.749501303Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.749579435Z namespaces/openshift/batch/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.749660907Z namespaces/openshift/batch/cronjobs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.74979614Z namespaces/openshift/batch/jobs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.749874593Z namespaces/openshift/build.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.749940884Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750076458Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750142529Z namespaces/openshift/core/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750224691Z namespaces/openshift/core/configmaps.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750390986Z namespaces/openshift/core/endpoints.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750535509Z namespaces/openshift/core/events.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750660722Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750787376Z namespaces/openshift/core/pods.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.750927529Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.751114464Z namespaces/openshift/core/secrets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.751385951Z namespaces/openshift/core/services.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.751491923Z namespaces/openshift/discovery.k8s.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.751576475Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.751664187Z namespaces/openshift/image.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.7517502Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.753886693Z namespaces/openshift/k8s.ovn.org/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.753949105Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.754084768Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.7541645Z namespaces/openshift/monitoring.coreos.com/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.754235102Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.754338985Z namespaces/openshift/networking.k8s.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.754409876Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.754489298Z namespaces/openshift/policy/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.75457134Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.754653593Z namespaces/openshift/route.openshift.io/ [must-gather-dsz7b] OUT 2026-10-30T11:56:04.754724574Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-dsz7b] OUT 2026-10-30T11:56:04.807412687Z [must-gather-dsz7b] OUT 2026-10-30T11:56:04.807446438Z sent 1,384 bytes received 149,545 bytes 301,858.00 bytes/sec [must-gather-dsz7b] OUT 2026-10-30T11:56:04.807455459Z total size is 1,848,079 speedup is 12.24 [must-gather ] OUT 2026-10-30T11:56:04.869147398Z namespace/openshift-must-gather-26zjn deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 3a327ebf-8bb6-4092-bbf5-2613aef6a4e3 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing