go test ./test/e2e/ocp/ -v -timeout 30m go: downloading github.com/onsi/gomega v1.39.1 go: downloading github.com/google/go-cmp v0.7.0 Starting OCP e2e test suite === RUN TestTrainJobWithClusterTrainingRuntime trainjob_test.go:54: Timed out after 120.001s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/ocp/trainjob_test.go:53 with: Expected : Not Ready to equal : Ready client.go:122: Controller logs: 2026-07-16T03:50:14Z INFO setup starting manager 2026-07-16T03:50:14Z INFO controller-runtime.metrics Starting metrics server 2026-07-16T03:50:14Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} 2026-07-16T03:50:14Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} 2026-07-16T03:50:14Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *v1.ValidatingWebhookConfiguration"} 2026-07-16T03:50:14Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *v1alpha1.Trainer"} 2026-07-16T03:50:14Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *v1.Deployment"} 2026-07-16T03:50:14Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *v1.Service"} 2026-07-16T03:50:14Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *v1.ConfigMap"} 2026-07-16T03:50:15Z INFO Starting Controller {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer"} 2026-07-16T03:50:15Z INFO Starting workers {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "worker count": 1} 2026-07-16T03:50:56Z INFO Added finalizer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "40e0297e-942b-4b5f-ade7-57bd25f0cfa2"} 2026-07-16T03:50:56Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "40e0297e-942b-4b5f-ade7-57bd25f0cfa2", "namespace": "opendatahub"} 2026-07-16T03:50:56Z INFO Created namespace {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "40e0297e-942b-4b5f-ade7-57bd25f0cfa2", "namespace": "opendatahub"} 2026/07/16 03:50:57 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:50:57Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "40e0297e-942b-4b5f-ade7-57bd25f0cfa2", "count": 34} 2026-07-16T03:50:58Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "40e0297e-942b-4b5f-ade7-57bd25f0cfa2", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:50:58Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "4f39d06c-b688-43de-8939-6c3b694f472a", "namespace": "opendatahub"} 2026/07/16 03:50:59 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:50:59Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "4f39d06c-b688-43de-8939-6c3b694f472a", "count": 34} 2026-07-16T03:51:00Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "4f39d06c-b688-43de-8939-6c3b694f472a", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:00Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "da4662d5-a65e-4093-a9e8-a2801ea57f4c", "namespace": "opendatahub"} 2026/07/16 03:51:01 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:01Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "da4662d5-a65e-4093-a9e8-a2801ea57f4c", "count": 34} 2026-07-16T03:51:02Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "da4662d5-a65e-4093-a9e8-a2801ea57f4c", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:02Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe33cf93-f76e-470b-9a6c-5a6d4b73de00", "namespace": "opendatahub"} 2026/07/16 03:51:03 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:03Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe33cf93-f76e-470b-9a6c-5a6d4b73de00", "count": 34} 2026-07-16T03:51:04Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe33cf93-f76e-470b-9a6c-5a6d4b73de00", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:04Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ca731cee-c026-4ab9-bb40-b615fea9a0f6", "namespace": "opendatahub"} 2026/07/16 03:51:05 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:05Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ca731cee-c026-4ab9-bb40-b615fea9a0f6", "count": 34} 2026-07-16T03:51:05Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ca731cee-c026-4ab9-bb40-b615fea9a0f6", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:06Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "6e939b8f-e62b-40e0-a9f4-05f90da1616a", "namespace": "opendatahub"} 2026/07/16 03:51:07 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:07Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "6e939b8f-e62b-40e0-a9f4-05f90da1616a", "count": 34} 2026-07-16T03:51:07Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "6e939b8f-e62b-40e0-a9f4-05f90da1616a", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:08Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "49a1d454-4c8c-4dde-b2e8-883adb09e81b", "namespace": "opendatahub"} 2026/07/16 03:51:09 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:09Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "49a1d454-4c8c-4dde-b2e8-883adb09e81b", "count": 34} 2026-07-16T03:51:09Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "49a1d454-4c8c-4dde-b2e8-883adb09e81b", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:10Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c5b21e1f-ba65-4813-9d7e-1e82025bf66c", "namespace": "opendatahub"} 2026/07/16 03:51:11 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:11Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c5b21e1f-ba65-4813-9d7e-1e82025bf66c", "count": 34} 2026-07-16T03:51:11Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c5b21e1f-ba65-4813-9d7e-1e82025bf66c", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:12Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "a5e2ac11-2e69-4da9-a518-2a78aa0f6a20", "namespace": "opendatahub"} 2026/07/16 03:51:13 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:13Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "a5e2ac11-2e69-4da9-a518-2a78aa0f6a20", "count": 34} 2026-07-16T03:51:14Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "a5e2ac11-2e69-4da9-a518-2a78aa0f6a20", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:15Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "9a13a9d6-8b97-4f9c-b73f-00acb20c8bcf", "namespace": "opendatahub"} 2026/07/16 03:51:16 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:16Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "9a13a9d6-8b97-4f9c-b73f-00acb20c8bcf", "count": 34} 2026-07-16T03:51:17Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "9a13a9d6-8b97-4f9c-b73f-00acb20c8bcf", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:20Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "137247f0-618b-4645-b695-0aaa98dec4d6", "namespace": "opendatahub"} 2026/07/16 03:51:21 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:21Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "137247f0-618b-4645-b695-0aaa98dec4d6", "count": 34} 2026-07-16T03:51:22Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "137247f0-618b-4645-b695-0aaa98dec4d6", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:27Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "55e964af-9b72-4c14-93ce-0ec74f0e0ff9", "namespace": "opendatahub"} 2026/07/16 03:51:28 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:28Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "55e964af-9b72-4c14-93ce-0ec74f0e0ff9", "count": 34} 2026-07-16T03:51:29Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "55e964af-9b72-4c14-93ce-0ec74f0e0ff9", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:51:39Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c5252613-350c-46e8-9888-0659016365e6", "namespace": "opendatahub"} 2026/07/16 03:51:40 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:51:40Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c5252613-350c-46e8-9888-0659016365e6", "count": 34} 2026-07-16T03:51:41Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c5252613-350c-46e8-9888-0659016365e6", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:52:01Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "71c3e687-a2b7-4517-9c52-d286bff6fe3c", "namespace": "opendatahub"} 2026/07/16 03:52:02 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:52:02Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "71c3e687-a2b7-4517-9c52-d286bff6fe3c", "count": 34} 2026-07-16T03:52:03Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "71c3e687-a2b7-4517-9c52-d286bff6fe3c", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:52:44Z INFO Reconciling Trainer {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "bf2d5c54-d443-43e7-88e4-62465704003e", "namespace": "opendatahub"} 2026/07/16 03:52:45 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-07-16T03:52:45Z INFO Applying manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "bf2d5c54-d443-43e7-88e4-62465704003e", "count": 34} 2026-07-16T03:52:46Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "bf2d5c54-d443-43e7-88e4-62465704003e", "error": "applying Trainer manifests: applying manifests: applying ClusterRole /kubeflow-trainer-controller-manager: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole kubeflow-trainer-controller-manager: clusterroles.rbac.authorization.k8s.io \"kubeflow-trainer-controller-manager\" is forbidden: user \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:trainer-operator-system\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"config.openshift.io\"], Resources:[\"apiservers\"], Verbs:[\"get\"]}"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-16T03:52:56Z INFO Trainer is being deleted, cleaning up managed resources {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "77eea1a2-de05-4a84-99f8-670724a4ff63"} client.go:139: Kubernetes events: 0001-01-01T00:00:00Z Normal Scheduled trainer-operator-controller-manager-68b698646c-52b94 Successfully assigned trainer-operator-system/trainer-operator-controller-manager-68b698646c-52b94 to ip-10-0-141-197.ec2.internal 2026-07-16T03:50:11Z Normal SuccessfulCreate trainer-operator-controller-manager-68b698646c Created pod: trainer-operator-controller-manager-68b698646c-52b94 2026-07-16T03:50:11Z Normal ScalingReplicaSet trainer-operator-controller-manager Scaled up replica set trainer-operator-controller-manager-68b698646c from 0 to 1 2026-07-16T03:50:12Z Normal AddedInterface trainer-operator-controller-manager-68b698646c-52b94 Add eth0 [10.132.0.29/23] from ovn-kubernetes 2026-07-16T03:50:12Z Normal Pulling trainer-operator-controller-manager-68b698646c-52b94 Pulling image "quay.io/opendatahub/odh-trainer-operator@sha256:1333d9ffea16b9ad03f64cfa0ea63b0130d7b2ca36d1897801418df26dd5bd5d" 2026-07-16T03:50:14Z Normal Pulled trainer-operator-controller-manager-68b698646c-52b94 Successfully pulled image "quay.io/opendatahub/odh-trainer-operator@sha256:1333d9ffea16b9ad03f64cfa0ea63b0130d7b2ca36d1897801418df26dd5bd5d" in 2.307s (2.307s including waiting). Image size: 197400766 bytes. 2026-07-16T03:50:14Z Normal Created trainer-operator-controller-manager-68b698646c-52b94 Created container: manager 2026-07-16T03:50:14Z Normal Started trainer-operator-controller-manager-68b698646c-52b94 Started container manager --- FAIL: TestTrainJobWithClusterTrainingRuntime (120.06s) FAIL FAIL github.com/opendatahub-io/trainer-operator/test/e2e/ocp 120.114s FAIL make: *** [Makefile:151: test-e2e-ocp] Error 1