[must-gather ] OUT 2026-11-04T08:28:19.678087894Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 14ccf5d7-71cf-4f73-bbbe-02ddb457322c ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-11-04T08:28:19.820262026Z namespace/openshift-must-gather-k42r2 created [must-gather ] OUT 2026-11-04T08:28:19.845826112Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-x5fxs created [must-gather ] OUT 2026-11-04T08:28:19.895496766Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-4b9lc] OUT 2026-11-04T08:29:53.129181865Z waiting for gather to complete [must-gather-4b9lc] OUT 2026-11-04T08:29:53.145009648Z downloading gather output [must-gather-4b9lc] OUT 2026-11-04T08:29:53.680478915Z receiving incremental file list [must-gather-4b9lc] OUT 2026-11-04T08:29:53.737040257Z ./ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.737230872Z aggregated-discovery-api.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.73792189Z aggregated-discovery-apis.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.751796521Z event-filter.html [must-gather-4b9lc] OUT 2026-11-04T08:29:53.752111369Z timestamp [must-gather-4b9lc] OUT 2026-11-04T08:29:53.752306875Z version [must-gather-4b9lc] OUT 2026-11-04T08:29:53.75287938Z namespaces/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.75288952Z namespaces/opendatahub/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.752974263Z namespaces/opendatahub/opendatahub.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753112936Z namespaces/opendatahub/apps.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753172668Z namespaces/opendatahub/apps.openshift.io/deploymentconfigs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.75325471Z namespaces/opendatahub/apps/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753307312Z namespaces/opendatahub/apps/daemonsets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753428485Z namespaces/opendatahub/apps/deployments.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753585209Z namespaces/opendatahub/apps/replicasets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753746683Z namespaces/opendatahub/apps/statefulsets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753791804Z namespaces/opendatahub/autoscaling/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753845776Z namespaces/opendatahub/autoscaling/horizontalpodautoscalers.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753897837Z namespaces/opendatahub/batch/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.753951199Z namespaces/opendatahub/batch/cronjobs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754041701Z namespaces/opendatahub/batch/jobs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754113483Z namespaces/opendatahub/build.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754159164Z namespaces/opendatahub/build.openshift.io/buildconfigs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754260387Z namespaces/opendatahub/build.openshift.io/builds.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754309518Z namespaces/opendatahub/core/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.75436289Z namespaces/opendatahub/core/configmaps.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754516064Z namespaces/opendatahub/core/endpoints.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754612696Z namespaces/opendatahub/core/events.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754749Z namespaces/opendatahub/core/persistentvolumeclaims.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754837892Z namespaces/opendatahub/core/pods.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.754981906Z namespaces/opendatahub/core/replicationcontrollers.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.755145161Z namespaces/opendatahub/core/secrets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.755337086Z namespaces/opendatahub/core/services.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.755409888Z namespaces/opendatahub/discovery.k8s.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.755464509Z namespaces/opendatahub/discovery.k8s.io/endpointslices.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.755525471Z namespaces/opendatahub/image.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.755583392Z namespaces/opendatahub/image.openshift.io/imagestreams.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.766761331Z namespaces/opendatahub/k8s.ovn.org/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.766835573Z namespaces/opendatahub/k8s.ovn.org/egressfirewalls.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.766982277Z namespaces/opendatahub/k8s.ovn.org/egressqoses.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767050149Z namespaces/opendatahub/monitoring.coreos.com/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767150751Z namespaces/opendatahub/monitoring.coreos.com/servicemonitors.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767238464Z namespaces/opendatahub/networking.k8s.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767310706Z namespaces/opendatahub/networking.k8s.io/networkpolicies.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767417178Z namespaces/opendatahub/pods/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767431479Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.76749091Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767629764Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/manager/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767642675Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/manager/manager/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767649565Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/manager/manager/logs/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767705516Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/manager/manager/logs/current.log [must-gather-4b9lc] OUT 2026-11-04T08:29:53.767919712Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/manager/manager/logs/previous.insecure.log [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768012544Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-96s9c/manager/manager/logs/previous.log [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768066456Z namespaces/opendatahub/policy/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768145288Z namespaces/opendatahub/policy/poddisruptionbudgets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768202839Z namespaces/opendatahub/route.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768260381Z namespaces/opendatahub/route.openshift.io/routes.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768316322Z namespaces/openshift/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768383104Z namespaces/openshift/openshift.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768441566Z namespaces/openshift/apps.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768501817Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768554639Z namespaces/openshift/apps/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768616121Z namespaces/openshift/apps/daemonsets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768709883Z namespaces/openshift/apps/deployments.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768810006Z namespaces/openshift/apps/replicasets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.768908648Z namespaces/openshift/apps/statefulsets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.76896159Z namespaces/openshift/autoscaling/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769020551Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769092143Z namespaces/openshift/batch/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769150345Z namespaces/openshift/batch/cronjobs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769265828Z namespaces/openshift/batch/jobs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769319509Z namespaces/openshift/build.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769381151Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769478193Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769533555Z namespaces/openshift/core/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769588307Z namespaces/openshift/core/configmaps.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.76971683Z namespaces/openshift/core/endpoints.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769815762Z namespaces/openshift/core/events.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.769909975Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.770010398Z namespaces/openshift/core/pods.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.770127141Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.770287175Z namespaces/openshift/core/secrets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.7704741Z namespaces/openshift/core/services.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.770536472Z namespaces/openshift/discovery.k8s.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.770593463Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.770652995Z namespaces/openshift/image.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.770722037Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.772740461Z namespaces/openshift/k8s.ovn.org/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.772800352Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.772921786Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.772980257Z namespaces/openshift/monitoring.coreos.com/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.773039089Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.773115121Z namespaces/openshift/networking.k8s.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.773176152Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.773248864Z namespaces/openshift/policy/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.773306826Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.773362427Z namespaces/openshift/route.openshift.io/ [must-gather-4b9lc] OUT 2026-11-04T08:29:53.773420859Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-4b9lc] OUT 2026-11-04T08:29:53.826314752Z [must-gather-4b9lc] OUT 2026-11-04T08:29:53.826365744Z sent 1,380 bytes received 72,793 bytes 148,346.00 bytes/sec [must-gather-4b9lc] OUT 2026-11-04T08:29:53.826377464Z total size is 874,786 speedup is 11.79 [must-gather ] OUT 2026-11-04T08:29:53.888100983Z namespace/openshift-must-gather-k42r2 deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 14ccf5d7-71cf-4f73-bbbe-02ddb457322c ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing