[must-gather ] OUT 2026-11-03T03:56:10.387237962Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 3f4ff0d6-b82e-4c4b-abba-28340c570a06 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing [must-gather ] OUT 2026-11-03T03:56:10.518760335Z namespace/openshift-must-gather-7kq9j created [must-gather ] OUT 2026-11-03T03:56:10.538656663Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-mbqdd created [must-gather ] OUT 2026-11-03T03:56:10.581208546Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather-zgzg4] OUT 2026-11-03T03:57:09.15462745Z waiting for gather to complete [must-gather-zgzg4] OUT 2026-11-03T03:57:09.169206518Z downloading gather output [must-gather-zgzg4] OUT 2026-11-03T03:57:09.542841553Z receiving incremental file list [must-gather-zgzg4] OUT 2026-11-03T03:57:09.579693092Z ./ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.579795584Z aggregated-discovery-api.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.580192412Z aggregated-discovery-apis.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.581392238Z event-filter.html [must-gather-zgzg4] OUT 2026-11-03T03:57:09.58151447Z timestamp [must-gather-zgzg4] OUT 2026-11-03T03:57:09.593433412Z version [must-gather-zgzg4] OUT 2026-11-03T03:57:09.593564485Z namespaces/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.593574785Z namespaces/opendatahub/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.593666757Z namespaces/opendatahub/opendatahub.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.59378349Z namespaces/opendatahub/apps.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.593859771Z namespaces/opendatahub/apps.openshift.io/deploymentconfigs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.593932873Z namespaces/opendatahub/apps/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594008714Z namespaces/opendatahub/apps/daemonsets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594126587Z namespaces/opendatahub/apps/deployments.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594302631Z namespaces/opendatahub/apps/replicasets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594443893Z namespaces/opendatahub/apps/statefulsets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594498875Z namespaces/opendatahub/autoscaling/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594581247Z namespaces/opendatahub/autoscaling/horizontalpodautoscalers.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594643528Z namespaces/opendatahub/batch/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594705529Z namespaces/opendatahub/batch/cronjobs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594822101Z namespaces/opendatahub/batch/jobs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594890943Z namespaces/opendatahub/build.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.594955274Z namespaces/opendatahub/build.openshift.io/buildconfigs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.595071267Z namespaces/opendatahub/build.openshift.io/builds.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.595139168Z namespaces/opendatahub/core/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.59521694Z namespaces/opendatahub/core/configmaps.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.595373493Z namespaces/opendatahub/core/endpoints.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.595494316Z namespaces/opendatahub/core/events.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.595664089Z namespaces/opendatahub/core/persistentvolumeclaims.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.595773842Z namespaces/opendatahub/core/pods.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.595948435Z namespaces/opendatahub/core/replicationcontrollers.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.596115819Z namespaces/opendatahub/core/secrets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.596412955Z namespaces/opendatahub/core/services.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.596479947Z namespaces/opendatahub/discovery.k8s.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.596542838Z namespaces/opendatahub/discovery.k8s.io/endpointslices.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.59661927Z namespaces/opendatahub/image.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.596685311Z namespaces/opendatahub/image.openshift.io/imagestreams.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607266604Z namespaces/opendatahub/k8s.ovn.org/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607319946Z namespaces/opendatahub/k8s.ovn.org/egressfirewalls.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607430098Z namespaces/opendatahub/k8s.ovn.org/egressqoses.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.60750673Z namespaces/opendatahub/monitoring.coreos.com/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.60755181Z namespaces/opendatahub/monitoring.coreos.com/servicemonitors.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607639032Z namespaces/opendatahub/networking.k8s.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607692784Z namespaces/opendatahub/networking.k8s.io/networkpolicies.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607777295Z namespaces/opendatahub/pods/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607784615Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607851837Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607953779Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/manager/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607963729Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/manager/manager/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.607970099Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/manager/manager/logs/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.608024711Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/manager/manager/logs/current.log [must-gather-zgzg4] OUT 2026-11-03T03:57:09.622612699Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/manager/manager/logs/previous.insecure.log [must-gather-zgzg4] OUT 2026-11-03T03:57:09.622701151Z namespaces/opendatahub/pods/kubeflow-trainer-controller-manager-774cfdb6cc-tsx9h/manager/manager/logs/previous.log [must-gather-zgzg4] OUT 2026-11-03T03:57:09.622748652Z namespaces/opendatahub/policy/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.622792453Z namespaces/opendatahub/policy/poddisruptionbudgets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.622850174Z namespaces/opendatahub/route.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.622905845Z namespaces/opendatahub/route.openshift.io/routes.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.622957146Z namespaces/openshift/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623022948Z namespaces/openshift/openshift.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623077739Z namespaces/openshift/apps.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.62313237Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623214832Z namespaces/openshift/apps/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623269433Z namespaces/openshift/apps/daemonsets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623383715Z namespaces/openshift/apps/deployments.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623496227Z namespaces/openshift/apps/replicasets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.62361971Z namespaces/openshift/apps/statefulsets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623692471Z namespaces/openshift/autoscaling/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623761083Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623839825Z namespaces/openshift/batch/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.623910326Z namespaces/openshift/batch/cronjobs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624036799Z namespaces/openshift/batch/jobs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.62410428Z namespaces/openshift/build.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624169822Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624317125Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624382526Z namespaces/openshift/core/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624473238Z namespaces/openshift/core/configmaps.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624609461Z namespaces/openshift/core/endpoints.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624733173Z namespaces/openshift/core/events.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624855286Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.624987719Z namespaces/openshift/core/pods.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.625099561Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.625318876Z namespaces/openshift/core/secrets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.625558581Z namespaces/openshift/core/services.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.625637473Z namespaces/openshift/discovery.k8s.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.625708434Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.625785156Z namespaces/openshift/image.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.625848437Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.627631135Z namespaces/openshift/k8s.ovn.org/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.627697436Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.627821329Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.62788785Z namespaces/openshift/monitoring.coreos.com/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.627955272Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.628022633Z namespaces/openshift/networking.k8s.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.628088595Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.628155506Z namespaces/openshift/policy/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.628244878Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.628313079Z namespaces/openshift/route.openshift.io/ [must-gather-zgzg4] OUT 2026-11-03T03:57:09.628378691Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather-zgzg4] OUT 2026-11-03T03:57:09.661904429Z [must-gather-zgzg4] OUT 2026-11-03T03:57:09.661926659Z sent 1,380 bytes received 150,041 bytes 302,842.00 bytes/sec [must-gather-zgzg4] OUT 2026-11-03T03:57:09.661932979Z total size is 1,906,629 speedup is 12.59 [must-gather ] OUT 2026-11-03T03:57:09.727224879Z namespace/openshift-must-gather-7kq9j deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 3f4ff0d6-b82e-4c4b-abba-28340c570a06 ClientVersion: 4.22.15 ClusterVersion: Stable at "4.22.20" ClusterOperators: clusteroperator/cloud-credential is not upgradeable because Upgradeable annotation cloudcredential.openshift.io/upgradeable-to on cloudcredential.operator.openshift.io/cluster object needs updating before upgrade. See Manually Creating IAM documentation for instructions on preparing a cluster for upgrade. clusteroperator/authentication is missing clusteroperator/cluster-autoscaler is missing clusteroperator/config-operator is missing clusteroperator/etcd is missing clusteroperator/machine-api is missing clusteroperator/machine-approver is missing clusteroperator/machine-config is missing clusteroperator/marketplace is missing