[must-gather ] OUT 2026-10-23T03:59:15.437095018Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 9afd7161-8df5-4bf8-bbca-4a613d7ec62c ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.18" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-10-23T03:59:15.551991783Z namespace/openshift-must-gather-vjnck created [must-gather ] OUT 2026-10-23T03:59:15.569425027Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-xqrhw created [must-gather ] OUT 2026-10-23T03:59:15.606042176Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-55dt7] OUT 2026-10-23T04:00:45.753882954Z waiting for gather to complete [must-gather-55dt7] OUT 2026-10-23T04:00:45.76717977Z downloading gather output [must-gather-55dt7] OUT 2026-10-23T04:00:46.313787517Z receiving incremental file list [must-gather-55dt7] OUT 2026-10-23T04:00:46.372390578Z ./ [must-gather-55dt7] OUT 2026-10-23T04:00:46.372629964Z aggregated-discovery-api.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.373116715Z aggregated-discovery-apis.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.374210491Z event-filter.html [must-gather-55dt7] OUT 2026-10-23T04:00:46.374340554Z timestamp [must-gather-55dt7] OUT 2026-10-23T04:00:46.384933206Z version [must-gather-55dt7] OUT 2026-10-23T04:00:46.398521008Z namespaces/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.398544259Z namespaces/openshift/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.398630391Z namespaces/openshift/openshift.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.398736093Z namespaces/openshift/apps.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.398803705Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.398867106Z namespaces/openshift/apps/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.398949459Z namespaces/openshift/apps/daemonsets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.399089782Z namespaces/openshift/apps/deployments.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.399207674Z namespaces/openshift/apps/replicasets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.399392059Z namespaces/openshift/apps/statefulsets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.39945786Z namespaces/openshift/autoscaling/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.399520232Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.399569583Z namespaces/openshift/batch/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.399630425Z namespaces/openshift/batch/cronjobs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.399723727Z namespaces/openshift/batch/jobs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.399785138Z namespaces/openshift/build.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.39985557Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.399967873Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.400033154Z namespaces/openshift/core/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.400128687Z namespaces/openshift/core/configmaps.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.40028939Z namespaces/openshift/core/endpoints.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.400399673Z namespaces/openshift/core/events.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.400515526Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.400623438Z namespaces/openshift/core/pods.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.400735131Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.400886365Z namespaces/openshift/core/secrets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.40110467Z namespaces/openshift/core/services.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.401178252Z namespaces/openshift/discovery.k8s.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.401243433Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.401312175Z namespaces/openshift/image.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.401376346Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.412975312Z namespaces/openshift/k8s.ovn.org/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.413048733Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.413241778Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.413296249Z namespaces/openshift/monitoring.coreos.com/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.413370651Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.413443943Z namespaces/openshift/networking.k8s.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.413500484Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.413571026Z namespaces/openshift/policy/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.413631397Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.413717929Z namespaces/openshift/route.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.41377498Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.413847552Z namespaces/redhat-ods-applications/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.413916904Z namespaces/redhat-ods-applications/redhat-ods-applications.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.414007376Z namespaces/redhat-ods-applications/apps.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.414076098Z namespaces/redhat-ods-applications/apps.openshift.io/deploymentconfigs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.41416476Z namespaces/redhat-ods-applications/apps/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.414226771Z namespaces/redhat-ods-applications/apps/daemonsets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.414340474Z namespaces/redhat-ods-applications/apps/deployments.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.414493248Z namespaces/redhat-ods-applications/apps/replicasets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.414632251Z namespaces/redhat-ods-applications/apps/statefulsets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.414699293Z namespaces/redhat-ods-applications/autoscaling/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.414766374Z namespaces/redhat-ods-applications/autoscaling/horizontalpodautoscalers.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.414836136Z namespaces/redhat-ods-applications/batch/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.414900207Z namespaces/redhat-ods-applications/batch/cronjobs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.41501044Z namespaces/redhat-ods-applications/batch/jobs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.415093842Z namespaces/redhat-ods-applications/build.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.415159363Z namespaces/redhat-ods-applications/build.openshift.io/buildconfigs.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.415273106Z namespaces/redhat-ods-applications/build.openshift.io/builds.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.415339838Z namespaces/redhat-ods-applications/core/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.415407279Z namespaces/redhat-ods-applications/core/configmaps.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.415570113Z namespaces/redhat-ods-applications/core/endpoints.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.415703506Z namespaces/redhat-ods-applications/core/events.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.41586067Z namespaces/redhat-ods-applications/core/persistentvolumeclaims.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.415969392Z namespaces/redhat-ods-applications/core/pods.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.416151867Z namespaces/redhat-ods-applications/core/replicationcontrollers.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.416312501Z namespaces/redhat-ods-applications/core/secrets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.416528036Z namespaces/redhat-ods-applications/core/services.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.416614188Z namespaces/redhat-ods-applications/discovery.k8s.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.41668066Z namespaces/redhat-ods-applications/discovery.k8s.io/endpointslices.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.416756111Z namespaces/redhat-ods-applications/image.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.416815773Z namespaces/redhat-ods-applications/image.openshift.io/imagestreams.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.416959286Z namespaces/redhat-ods-applications/k8s.ovn.org/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417015197Z namespaces/redhat-ods-applications/k8s.ovn.org/egressfirewalls.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.417151361Z namespaces/redhat-ods-applications/k8s.ovn.org/egressqoses.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.417220672Z namespaces/redhat-ods-applications/monitoring.coreos.com/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417284644Z namespaces/redhat-ods-applications/monitoring.coreos.com/servicemonitors.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.417360685Z namespaces/redhat-ods-applications/networking.k8s.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417420977Z namespaces/redhat-ods-applications/networking.k8s.io/networkpolicies.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.417502029Z namespaces/redhat-ods-applications/pods/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417509069Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417576991Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/kubeflow-trainer-controller-manager-5695f7f854-fp28q.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.417697764Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/manager/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417706444Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/manager/manager/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417711504Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/manager/manager/logs/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.417783146Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/manager/manager/logs/current.log [must-gather-55dt7] OUT 2026-10-23T04:00:46.42850758Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/manager/manager/logs/previous.insecure.log [must-gather-55dt7] OUT 2026-10-23T04:00:46.428612703Z namespaces/redhat-ods-applications/pods/kubeflow-trainer-controller-manager-5695f7f854-fp28q/manager/manager/logs/previous.log [must-gather-55dt7] OUT 2026-10-23T04:00:46.428675474Z namespaces/redhat-ods-applications/policy/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.428758696Z namespaces/redhat-ods-applications/policy/poddisruptionbudgets.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.428813808Z namespaces/redhat-ods-applications/route.openshift.io/ [must-gather-55dt7] OUT 2026-10-23T04:00:46.428888229Z namespaces/redhat-ods-applications/route.openshift.io/routes.yaml [must-gather-55dt7] OUT 2026-10-23T04:00:46.484221863Z [must-gather-55dt7] OUT 2026-10-23T04:00:46.484262184Z sent 1,380 bytes received 166,409 bytes 111,859.33 bytes/sec [must-gather-55dt7] OUT 2026-10-23T04:00:46.484269274Z total size is 2,048,977 speedup is 12.21 [must-gather ] OUT 2026-10-23T04:00:46.544929414Z namespace/openshift-must-gather-vjnck deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 9afd7161-8df5-4bf8-bbca-4a613d7ec62c ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.18" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing