[must-gather ] OUT 2026-10-30T21:26:31.906199977Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 9ee1c926-3dd1-4e20-a5f9-3a1366240021 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-10-30T21:26:32.04380764Z namespace/openshift-must-gather-zpr7c created [must-gather ] OUT 2026-10-30T21:26:32.06850236Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-txf8v created [must-gather ] OUT 2026-10-30T21:26:32.115526833Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-hqn8f] OUT 2026-10-30T21:27:43.150698999Z waiting for gather to complete [must-gather-hqn8f] OUT 2026-10-30T21:27:43.165896375Z downloading gather output [must-gather-hqn8f] OUT 2026-10-30T21:27:43.605735401Z receiving incremental file list [must-gather-hqn8f] OUT 2026-10-30T21:27:43.639291195Z ./ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.639427675Z aggregated-discovery-api.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.639882439Z aggregated-discovery-apis.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.640498124Z event-filter.html [must-gather-hqn8f] OUT 2026-10-30T21:27:43.640629234Z timestamp [must-gather-hqn8f] OUT 2026-10-30T21:27:43.6520583Z version [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652220802Z namespaces/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652251654Z namespaces/opendatahub/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652341351Z namespaces/opendatahub/opendatahub.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652446899Z namespaces/opendatahub/apps.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652522455Z namespaces/opendatahub/apps.openshift.io/deploymentconfigs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652586559Z namespaces/opendatahub/apps/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652660055Z namespaces/opendatahub/apps/daemonsets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652754712Z namespaces/opendatahub/apps/deployments.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.652894092Z namespaces/opendatahub/apps/replicasets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653022952Z namespaces/opendatahub/apps/statefulsets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653077385Z namespaces/opendatahub/autoscaling/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653169812Z namespaces/opendatahub/autoscaling/horizontalpodautoscalers.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653207075Z namespaces/opendatahub/batch/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.65327038Z namespaces/opendatahub/batch/cronjobs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653363867Z namespaces/opendatahub/batch/jobs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653419581Z namespaces/opendatahub/build.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653477715Z namespaces/opendatahub/build.openshift.io/buildconfigs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653567002Z namespaces/opendatahub/build.openshift.io/builds.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653621306Z namespaces/opendatahub/core/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.65367412Z namespaces/opendatahub/core/configmaps.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653839632Z namespaces/opendatahub/core/endpoints.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.653940019Z namespaces/opendatahub/core/events.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.6540826Z namespaces/opendatahub/core/persistentvolumeclaims.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.654202959Z namespaces/opendatahub/core/pods.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.65434777Z namespaces/opendatahub/core/replicationcontrollers.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.65448766Z namespaces/opendatahub/core/secrets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.654691245Z namespaces/opendatahub/core/services.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.654769901Z namespaces/opendatahub/discovery.k8s.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.654831585Z namespaces/opendatahub/discovery.k8s.io/endpointslices.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.6548981Z namespaces/opendatahub/image.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.654954455Z namespaces/opendatahub/image.openshift.io/imagestreams.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.664783112Z namespaces/opendatahub/k8s.ovn.org/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.664850047Z namespaces/opendatahub/k8s.ovn.org/egressfirewalls.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.664975826Z namespaces/opendatahub/k8s.ovn.org/egressqoses.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665037981Z namespaces/opendatahub/monitoring.coreos.com/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665107476Z namespaces/opendatahub/monitoring.coreos.com/servicemonitors.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665202523Z namespaces/opendatahub/networking.k8s.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665274938Z namespaces/opendatahub/networking.k8s.io/networkpolicies.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665364575Z namespaces/opendatahub/pods/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665373276Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665452732Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.66556226Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/manager/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.66556974Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/manager/manager/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665574621Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/manager/manager/logs/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.665624864Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/manager/manager/logs/current.log [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679389593Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/manager/manager/logs/previous.insecure.log [must-gather-hqn8f] OUT 2026-10-30T21:27:43.67947926Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84b74c44f8-j6kjt/manager/manager/logs/previous.log [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679505692Z namespaces/opendatahub/policy/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679588628Z namespaces/opendatahub/policy/poddisruptionbudgets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679649803Z namespaces/opendatahub/route.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679704536Z namespaces/opendatahub/route.openshift.io/routes.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.67975458Z namespaces/openshift/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679820435Z namespaces/openshift/openshift.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679877669Z namespaces/openshift/apps.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679917162Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.679983067Z namespaces/openshift/apps/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.68002253Z namespaces/openshift/apps/daemonsets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680149849Z namespaces/openshift/apps/deployments.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680242286Z namespaces/openshift/apps/replicasets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680336433Z namespaces/openshift/apps/statefulsets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680391067Z namespaces/openshift/autoscaling/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680441001Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680495595Z namespaces/openshift/batch/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680543238Z namespaces/openshift/batch/cronjobs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680637076Z namespaces/openshift/batch/jobs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680681729Z namespaces/openshift/build.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680742343Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.68082961Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680885204Z namespaces/openshift/core/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.680942478Z namespaces/openshift/core/configmaps.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681066717Z namespaces/openshift/core/endpoints.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681177996Z namespaces/openshift/core/events.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681271992Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681369Z namespaces/openshift/core/pods.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681460076Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681613268Z namespaces/openshift/core/secrets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681813272Z namespaces/openshift/core/services.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681869587Z namespaces/openshift/discovery.k8s.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681923271Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.681979095Z namespaces/openshift/image.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.682035799Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.683651089Z namespaces/openshift/k8s.ovn.org/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.683709333Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.68380879Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.683860074Z namespaces/openshift/monitoring.coreos.com/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.683916218Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.683967882Z namespaces/openshift/networking.k8s.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.684024096Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.68407372Z namespaces/openshift/policy/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.684134104Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.684198509Z namespaces/openshift/route.openshift.io/ [must-gather-hqn8f] OUT 2026-10-30T21:27:43.684253993Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-hqn8f] OUT 2026-10-30T21:27:43.715264529Z [must-gather-hqn8f] OUT 2026-10-30T21:27:43.715390118Z sent 1,380 bytes received 154,546 bytes 311,852.00 bytes/sec [must-gather-hqn8f] OUT 2026-10-30T21:27:43.715465634Z total size is 1,955,448 speedup is 12.54 [must-gather ] OUT 2026-10-30T21:27:43.77769658Z namespace/openshift-must-gather-zpr7c deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 9ee1c926-3dd1-4e20-a5f9-3a1366240021 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing