[must-gather ] OUT 2026-11-04T04:25:53.478179506Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: f25fc1d5-367a-4a14-ad64-dbfc54c3b980 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-11-04T04:25:53.613508211Z namespace/openshift-must-gather-m8rth created [must-gather ] OUT 2026-11-04T04:25:53.632033026Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-bxvm5 created [must-gather ] OUT 2026-11-04T04:25:53.678360725Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-gqqhj] OUT 2026-11-04T04:29:21.679438375Z waiting for gather to complete [must-gather-gqqhj] OUT 2026-11-04T04:29:21.69404505Z downloading gather output [must-gather-gqqhj] OUT 2026-11-04T04:29:22.74785639Z receiving incremental file list [must-gather-gqqhj] OUT 2026-11-04T04:29:22.93758997Z ./ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.937753482Z aggregated-discovery-api.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.9383285Z aggregated-discovery-apis.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.950294178Z event-filter.html [must-gather-gqqhj] OUT 2026-11-04T04:29:22.95044453Z timestamp [must-gather-gqqhj] OUT 2026-11-04T04:29:22.950578092Z version [must-gather-gqqhj] OUT 2026-11-04T04:29:22.993378042Z namespaces/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.993415732Z namespaces/openshift/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.993451113Z namespaces/openshift/openshift.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.993582265Z namespaces/openshift/apps.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.993661726Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.993760717Z namespaces/openshift/apps/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.993844588Z namespaces/openshift/apps/daemonsets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.99398338Z namespaces/openshift/apps/deployments.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994102252Z namespaces/openshift/apps/replicasets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994216423Z namespaces/openshift/apps/statefulsets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994276264Z namespaces/openshift/autoscaling/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994349185Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994427996Z namespaces/openshift/batch/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994489457Z namespaces/openshift/batch/cronjobs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994605109Z namespaces/openshift/batch/jobs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.99467398Z namespaces/openshift/build.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994738971Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994862432Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.994947144Z namespaces/openshift/core/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.995010565Z namespaces/openshift/core/configmaps.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.995173507Z namespaces/openshift/core/endpoints.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.995297769Z namespaces/openshift/core/events.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.99541245Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.995526082Z namespaces/openshift/core/pods.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.995638213Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.995801456Z namespaces/openshift/core/secrets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.996019489Z namespaces/openshift/core/services.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.9960958Z namespaces/openshift/discovery.k8s.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.996161831Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:22.996237452Z namespaces/openshift/image.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:22.996299273Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.006924172Z namespaces/openshift/k8s.ovn.org/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.006993042Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007125844Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007180835Z namespaces/openshift/monitoring.coreos.com/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007266126Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007339037Z namespaces/openshift/networking.k8s.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007405118Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007483209Z namespaces/openshift/policy/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.0075454Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007626751Z namespaces/openshift/route.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007689512Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007765283Z namespaces/redhat-ods-applications/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007832974Z namespaces/redhat-ods-applications/redhat-ods-applications.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.007952106Z namespaces/redhat-ods-applications/apps.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008013347Z namespaces/redhat-ods-applications/apps.openshift.io/deploymentconfigs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008091128Z namespaces/redhat-ods-applications/apps/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008159739Z namespaces/redhat-ods-applications/apps/daemonsets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.00827814Z namespaces/redhat-ods-applications/apps/deployments.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008431593Z namespaces/redhat-ods-applications/apps/replicasets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008586385Z namespaces/redhat-ods-applications/apps/statefulsets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008657576Z namespaces/redhat-ods-applications/autoscaling/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008735597Z namespaces/redhat-ods-applications/autoscaling/horizontalpodautoscalers.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008809208Z namespaces/redhat-ods-applications/batch/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.008894019Z namespaces/redhat-ods-applications/batch/cronjobs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009017831Z namespaces/redhat-ods-applications/batch/jobs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009085652Z namespaces/redhat-ods-applications/build.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009157363Z namespaces/redhat-ods-applications/build.openshift.io/buildconfigs.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009276014Z namespaces/redhat-ods-applications/build.openshift.io/builds.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009349075Z namespaces/redhat-ods-applications/core/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009410026Z namespaces/redhat-ods-applications/core/configmaps.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009571389Z namespaces/redhat-ods-applications/core/endpoints.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.00969499Z namespaces/redhat-ods-applications/core/events.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.009866553Z namespaces/redhat-ods-applications/core/persistentvolumeclaims.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010000225Z namespaces/redhat-ods-applications/core/pods.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010163657Z namespaces/redhat-ods-applications/core/replicationcontrollers.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010328379Z namespaces/redhat-ods-applications/core/secrets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010541032Z namespaces/redhat-ods-applications/core/services.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010629133Z namespaces/redhat-ods-applications/discovery.k8s.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010695024Z namespaces/redhat-ods-applications/discovery.k8s.io/endpointslices.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010786566Z namespaces/redhat-ods-applications/image.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.010852927Z namespaces/redhat-ods-applications/image.openshift.io/imagestreams.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011044619Z namespaces/redhat-ods-applications/k8s.ovn.org/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.01110873Z namespaces/redhat-ods-applications/k8s.ovn.org/egressfirewalls.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011231902Z namespaces/redhat-ods-applications/k8s.ovn.org/egressqoses.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011302913Z namespaces/redhat-ods-applications/monitoring.coreos.com/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011372104Z namespaces/redhat-ods-applications/monitoring.coreos.com/servicemonitors.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011455575Z namespaces/redhat-ods-applications/networking.k8s.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011512126Z namespaces/redhat-ods-applications/networking.k8s.io/networkpolicies.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011607797Z namespaces/redhat-ods-applications/pods/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011617027Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011684468Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.01180338Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/manager/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.01181321Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/manager/manager/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.01181855Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/manager/manager/logs/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.011900811Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/manager/manager/logs/current.log [must-gather-gqqhj] OUT 2026-11-04T04:29:23.046864091Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/manager/manager/logs/previous.insecure.log [must-gather-gqqhj] OUT 2026-11-04T04:29:23.047015594Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-6cbdc4f74b-6j5ks/manager/manager/logs/previous.log [must-gather-gqqhj] OUT 2026-11-04T04:29:23.047083754Z namespaces/redhat-ods-applications/policy/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.047168945Z namespaces/redhat-ods-applications/policy/poddisruptionbudgets.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.047247727Z namespaces/redhat-ods-applications/route.openshift.io/ [must-gather-gqqhj] OUT 2026-11-04T04:29:23.047330708Z namespaces/redhat-ods-applications/route.openshift.io/routes.yaml [must-gather-gqqhj] OUT 2026-11-04T04:29:23.182241599Z [must-gather-gqqhj] OUT 2026-11-04T04:29:23.182283429Z sent 1,380 bytes received 121,470 bytes 49,140.00 bytes/sec [must-gather-gqqhj] OUT 2026-11-04T04:29:23.18229301Z total size is 1,508,091 speedup is 12.28 [must-gather ] OUT 2026-11-04T04:29:23.285631708Z namespace/openshift-must-gather-m8rth deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: f25fc1d5-367a-4a14-ad64-dbfc54c3b980 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing