[must-gather ] OUT 2026-11-04T09:52:48.25233344Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a2e3c4a2-67ff-4fdd-bf96-6154fefc3a19 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-11-04T09:52:48.384820142Z namespace/openshift-must-gather-vbl9j created [must-gather ] OUT 2026-11-04T09:52:48.400697452Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-9qb4q created [must-gather ] OUT 2026-11-04T09:52:48.45702323Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-8lg4c] OUT 2026-11-04T09:56:17.213780911Z waiting for gather to complete [must-gather-8lg4c] OUT 2026-11-04T09:56:17.228050346Z downloading gather output [must-gather-8lg4c] OUT 2026-11-04T09:56:18.267781081Z receiving incremental file list [must-gather-8lg4c] OUT 2026-11-04T09:56:18.405379994Z ./ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.405442787Z aggregated-discovery-api.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.405851759Z aggregated-discovery-apis.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.42080043Z event-filter.html [must-gather-8lg4c] OUT 2026-11-04T09:56:18.420910456Z timestamp [must-gather-8lg4c] OUT 2026-11-04T09:56:18.421048283Z version [must-gather-8lg4c] OUT 2026-11-04T09:56:18.455779229Z namespaces/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.45579982Z namespaces/opendatahub/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.455917976Z namespaces/opendatahub/opendatahub.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456089365Z namespaces/opendatahub/apps.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456154919Z namespaces/opendatahub/apps.openshift.io/deploymentconfigs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456226222Z namespaces/opendatahub/apps/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456299156Z namespaces/opendatahub/apps/daemonsets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456450544Z namespaces/opendatahub/apps/deployments.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456602232Z namespaces/opendatahub/apps/replicasets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456725828Z namespaces/opendatahub/apps/statefulsets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456780181Z namespaces/opendatahub/autoscaling/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456842064Z namespaces/opendatahub/autoscaling/horizontalpodautoscalers.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456900147Z namespaces/opendatahub/batch/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.456968421Z namespaces/opendatahub/batch/cronjobs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457072107Z namespaces/opendatahub/batch/jobs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457120409Z namespaces/opendatahub/build.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457179762Z namespaces/opendatahub/build.openshift.io/buildconfigs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457282767Z namespaces/opendatahub/build.openshift.io/builds.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457342241Z namespaces/opendatahub/core/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457386773Z namespaces/opendatahub/core/configmaps.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457566052Z namespaces/opendatahub/core/endpoints.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457669308Z namespaces/opendatahub/core/events.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.457809585Z namespaces/opendatahub/core/persistentvolumeclaims.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.45790884Z namespaces/opendatahub/core/pods.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.458073589Z namespaces/opendatahub/core/replicationcontrollers.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.458214316Z namespaces/opendatahub/core/secrets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.458413707Z namespaces/opendatahub/core/services.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.458486841Z namespaces/opendatahub/discovery.k8s.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.458542073Z namespaces/opendatahub/discovery.k8s.io/endpointslices.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.458611057Z namespaces/opendatahub/image.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.4586612Z namespaces/opendatahub/image.openshift.io/imagestreams.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.470587013Z namespaces/opendatahub/k8s.ovn.org/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.470649926Z namespaces/opendatahub/k8s.ovn.org/egressfirewalls.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.470774823Z namespaces/opendatahub/k8s.ovn.org/egressqoses.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.470839046Z namespaces/opendatahub/monitoring.coreos.com/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.47091574Z namespaces/opendatahub/monitoring.coreos.com/servicemonitors.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471027386Z namespaces/opendatahub/networking.k8s.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.47110739Z namespaces/opendatahub/networking.k8s.io/networkpolicies.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471209186Z namespaces/opendatahub/pods/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471219676Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471279639Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/kubeflow-trainer-controller-manager-84d6d98cc8-tn767.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471407246Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/manager/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471417756Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/manager/manager/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471424237Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/manager/manager/logs/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471475359Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/manager/manager/logs/current.log [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471667379Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/manager/manager/logs/previous.insecure.log [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471773005Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-84d6d98cc8-tn767/manager/manager/logs/previous.log [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471794226Z namespaces/opendatahub/policy/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471854799Z namespaces/opendatahub/policy/poddisruptionbudgets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471903992Z namespaces/opendatahub/route.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.471970455Z namespaces/opendatahub/route.openshift.io/routes.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472099872Z namespaces/openshift/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472143724Z namespaces/openshift/openshift.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472214258Z namespaces/openshift/apps.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.47225422Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472318844Z namespaces/openshift/apps/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472358295Z namespaces/openshift/apps/daemonsets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472460841Z namespaces/openshift/apps/deployments.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472553546Z namespaces/openshift/apps/replicasets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472647111Z namespaces/openshift/apps/statefulsets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472702973Z namespaces/openshift/autoscaling/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472753486Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472808899Z namespaces/openshift/batch/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472860802Z namespaces/openshift/batch/cronjobs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.472980028Z namespaces/openshift/batch/jobs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473060772Z namespaces/openshift/build.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473120045Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.47321738Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473257302Z namespaces/openshift/core/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473320556Z namespaces/openshift/core/configmaps.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473431952Z namespaces/openshift/core/endpoints.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473520756Z namespaces/openshift/core/events.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473617452Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473709426Z namespaces/openshift/core/pods.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.473804881Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.47397495Z namespaces/openshift/core/secrets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.474164Z namespaces/openshift/core/services.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.474221163Z namespaces/openshift/discovery.k8s.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.474272366Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.474334229Z namespaces/openshift/image.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.474388032Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476483961Z namespaces/openshift/k8s.ovn.org/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476516673Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476629639Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476679901Z namespaces/openshift/monitoring.coreos.com/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476721034Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476784317Z namespaces/openshift/networking.k8s.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476825649Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476885562Z namespaces/openshift/policy/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.476928454Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.477018069Z namespaces/openshift/route.openshift.io/ [must-gather-8lg4c] OUT 2026-11-04T09:56:18.477063222Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-8lg4c] OUT 2026-11-04T09:56:18.610674056Z [must-gather-8lg4c] OUT 2026-11-04T09:56:18.610719019Z sent 1,380 bytes received 73,257 bytes 49,758.00 bytes/sec [must-gather-8lg4c] OUT 2026-11-04T09:56:18.6107358Z total size is 876,560 speedup is 11.74 [must-gather ] OUT 2026-11-04T09:56:18.711057004Z namespace/openshift-must-gather-vbl9j deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: a2e3c4a2-67ff-4fdd-bf96-6154fefc3a19 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing