go test ./test/e2e/upgrade/post/ -v -timeout 10m Starting post-upgrade smoke test suite === RUN TestPostUpgradeControllerRunning --- PASS: TestPostUpgradeControllerRunning (0.00s) === RUN TestPostUpgradeTrainerStillReady post_upgrade_test.go:81: Timed out after 300.000s. The function passed to Eventually failed at /workspace/source/src_code/test/e2e/upgrade/post/post_upgrade_test.go:64 with: Expected : Not Ready to equal : Ready client.go:120: Controller logs: 2026-09-07T03:00:42Z INFO setup starting manager 2026-09-07T03:00:42Z INFO controller-runtime.metrics Starting metrics server 2026-09-07T03:00:42Z INFO starting server {"name": "health probe", "addr": "[::]:8081"} 2026-09-07T03:00:42Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8443", "secure": true} 2026-09-07T03:00:42Z INFO Starting EventSource {"controller": "tlssecurityprofilewatcher", "source": "kind source: *v1.APIServer"} 2026-09-07T03:00:42Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *v1.ConfigMap"} 2026-09-07T03:00:42Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *unstructured.Unstructured"} 2026-09-07T03:00:42Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *unstructured.Unstructured"} 2026-09-07T03:00:42Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *unstructured.Unstructured"} 2026-09-07T03:00:42Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *unstructured.Unstructured"} 2026-09-07T03:00:42Z INFO Starting EventSource {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "source": "kind source: *unstructured.Unstructured"} 2026-09-07T03:00:42Z INFO Starting Controller {"controller": "tlssecurityprofilewatcher"} 2026-09-07T03:00:42Z INFO Starting workers {"controller": "tlssecurityprofilewatcher", "worker count": 1} 2026-09-07T03:00:43Z INFO Starting Controller {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer"} 2026-09-07T03:00:43Z INFO Starting workers {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "worker count": 1} 2026-09-07T03:00:43Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a"} 2026-09-07T03:00:43Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a"} 2026-09-07T03:00:43Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:43Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:43Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:43Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:44 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:44Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "count": 21} 2026/09/07 03:00:44 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:44Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "count": 14} 2026/09/07 03:00:44 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:44Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "count": 3} 2026-09-07T03:00:44Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:45Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "d7655e51-8b01-4492-8fb6-ceb40e7dc11a", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:45Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089"} 2026-09-07T03:00:45Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089"} 2026-09-07T03:00:45Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:45Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:45Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:45Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:45 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:46Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "count": 21} 2026/09/07 03:00:46 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:46Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "count": 14} 2026/09/07 03:00:46 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:46Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "count": 3} 2026-09-07T03:00:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:46Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7f3e2ec6-fc89-46ee-a8bb-1ac943207089", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:46Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0"} 2026-09-07T03:00:46Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0"} 2026-09-07T03:00:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:47 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:47Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "count": 21} 2026/09/07 03:00:47 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:47Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "count": 14} 2026/09/07 03:00:47 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:47Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "count": 3} 2026-09-07T03:00:47Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:47Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "fe84f500-e728-496d-b00e-edba25186bb0", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:47Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1"} 2026-09-07T03:00:47Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1"} 2026-09-07T03:00:47Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:47Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:47Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:47Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:48 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:48Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "count": 21} 2026/09/07 03:00:48 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:48Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "count": 14} 2026/09/07 03:00:48 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:48Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "count": 3} 2026-09-07T03:00:48Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:48Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "f8147b42-0459-4525-9ff7-44ac89b8a3e1", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:48Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff"} 2026-09-07T03:00:48Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff"} 2026-09-07T03:00:48Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:48Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:48Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:48Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:49 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:49Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "count": 21} 2026/09/07 03:00:49 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:49Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "count": 14} 2026/09/07 03:00:49 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:49Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "count": 3} 2026-09-07T03:00:49Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:49Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "3fbcfc52-a18e-4ab4-800c-e33f5ef0e2ff", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:49Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468"} 2026-09-07T03:00:49Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468"} 2026-09-07T03:00:49Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:50Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:50Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:50Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:50 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:50Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "count": 21} 2026/09/07 03:00:50 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:50Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "count": 14} 2026/09/07 03:00:50 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:50Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "count": 3} 2026-09-07T03:00:50Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:51Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "985bf2a9-4c0d-4a4b-959c-a7bb328dd468", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:51Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d"} 2026-09-07T03:00:51Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d"} 2026-09-07T03:00:51Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:51Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:51Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:51Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:52 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:52Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "count": 21} 2026/09/07 03:00:52 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:52Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "count": 14} 2026/09/07 03:00:52 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:52Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "count": 3} 2026-09-07T03:00:52Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:52Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8ea795e1-7764-4d24-900a-f19e21baef9d", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:52Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683"} 2026-09-07T03:00:52Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683"} 2026-09-07T03:00:52Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:52Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:52Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:52Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:53 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:53Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "count": 21} 2026/09/07 03:00:54 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:54Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "count": 14} 2026/09/07 03:00:54 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:54Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "count": 3} 2026-09-07T03:00:54Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:54Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c31951a0-e6b7-495f-8971-fd9f27860683", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:54Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889"} 2026-09-07T03:00:54Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889"} 2026-09-07T03:00:54Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:54Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:54Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:54Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:55 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:55Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "count": 21} 2026/09/07 03:00:55 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:55Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "count": 14} 2026/09/07 03:00:55 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:55Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "count": 3} 2026-09-07T03:00:55Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:56Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "c4d5bc54-fc24-4608-8ce2-85bcf9840889", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:00:57Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf"} 2026-09-07T03:00:57Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf"} 2026-09-07T03:00:57Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:00:57Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:00:57Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:00:57Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:00:58 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:58Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "count": 21} 2026/09/07 03:00:58 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:58Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "count": 14} 2026/09/07 03:00:58 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:00:58Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "count": 3} 2026-09-07T03:00:58Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:00:58Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "ad4c7d29-3384-4ddf-8190-c7feb8041cbf", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:01:01Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a"} 2026-09-07T03:01:01Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a"} 2026-09-07T03:01:01Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:01:01Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:01:01Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:01:01Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:01:02 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:02Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "count": 21} 2026/09/07 03:01:02 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:02Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "count": 14} 2026/09/07 03:01:02 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:02Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "count": 3} 2026-09-07T03:01:02Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:01:02Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "b39a1d64-433c-4abc-ba9a-aeb181e9581a", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:01:07Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972"} 2026-09-07T03:01:07Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972"} 2026-09-07T03:01:07Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:01:07Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:01:07Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:01:07Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:01:08 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:08Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "count": 21} 2026/09/07 03:01:08 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:08Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "count": 14} 2026/09/07 03:01:08 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:08Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "count": 3} 2026-09-07T03:01:08Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:01:08Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "873e162a-6693-43da-923d-dba7fa6ac972", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:01:18Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e"} 2026-09-07T03:01:18Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e"} 2026-09-07T03:01:18Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:01:18Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:01:18Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:01:18Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:01:19 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:19Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "count": 21} 2026/09/07 03:01:20 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:20Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "count": 14} 2026/09/07 03:01:20 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:20Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "count": 3} 2026-09-07T03:01:20Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:01:20Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "344c75a6-dbb8-4161-98d9-dd748753637e", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:01:40Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82"} 2026-09-07T03:01:40Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82"} 2026-09-07T03:01:40Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:01:40Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:01:40Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:01:40Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:01:41 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:41Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "count": 21} 2026/09/07 03:01:41 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:41Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "count": 14} 2026/09/07 03:01:41 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:01:41Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "count": 3} 2026-09-07T03:01:41Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:01:41Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "89fc2bb2-eeb4-4308-813b-14c50a0c5a82", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:02:22Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794"} 2026-09-07T03:02:22Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794"} 2026-09-07T03:02:22Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:02:22Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:02:22Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:02:22Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:02:23 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:02:23Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "count": 21} 2026/09/07 03:02:23 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:02:23Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "count": 14} 2026/09/07 03:02:23 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:02:23Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "count": 3} 2026-09-07T03:02:23Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:02:24Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "8e573d39-a3d1-4d7b-8604-df2de56c6794", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-09-07T03:03:45Z INFO reconcile {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da"} 2026-09-07T03:03:45Z INFO apply {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da"} 2026-09-07T03:03:45Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).checkDependencies-fm"} 2026-09-07T03:03:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).ensureNamespace-fm"} 2026-09-07T03:03:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).updateReleases-fm"} 2026-09-07T03:03:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm"} 2026/09/07 03:03:46 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:03:46Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered manifests {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "count": 21} 2026/09/07 03:03:46 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:03:46Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered runtimes {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "count": 14} 2026/09/07 03:03:46 Warning: Sorting order is set both in 'kustomization.yaml' ('sortOptions') and in a CLI flag ('--reorder'). Using the kustomization file over the CLI flag. 2026-09-07T03:03:46Z INFO action.github.com/opendatahub-io/trainer-operator/internal/controller.(*trainerActions).renderManifests-fm Rendered imagestreams {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "count": 3} 2026-09-07T03:03:46Z INFO Executing action {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "action": "github.com/opendatahub-io/trainer-operator/internal/controller.NewReconciler.withImmutableFieldRecovery.func11"} 2026-09-07T03:03:47Z ERROR Reconciler error {"controller": "trainer", "controllerGroup": "components.platform.opendatahub.io", "controllerKind": "Trainer", "Trainer": {"name":"default-trainer"}, "namespace": "", "name": "default-trainer", "reconcileID": "7263e6ea-092a-46ca-a394-5c8abd6293da", "error": "provisioning failed: failure deploying resource /trainer-tls-profile: apply failed rbac.authorization.k8s.io/v1, Kind=ClusterRole: unable to patch rbac.authorization.k8s.io/v1, Kind=ClusterRole trainer-tls-profile: clusterroles.rbac.authorization.k8s.io \"trainer-tls-profile\" is forbidden: User \"system:serviceaccount:trainer-operator-system:trainer-operator-controller-manager\" cannot patch resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 client.go:120: Kubernetes events: 0001-01-01T00:00:00Z Normal Scheduled trainer-operator-controller-manager-668cdd7dcf-7rp6f Successfully assigned trainer-operator-system/trainer-operator-controller-manager-668cdd7dcf-7rp6f to ip-10-0-133-123.ec2.internal 0001-01-01T00:00:00Z Normal Scheduled trainer-operator-controller-manager-7f578b678f-wsz2n Successfully assigned trainer-operator-system/trainer-operator-controller-manager-7f578b678f-wsz2n to ip-10-0-128-145.ec2.internal 2026-09-07T02:59:28Z Normal SuccessfulCreate trainer-operator-controller-manager-668cdd7dcf Created pod: trainer-operator-controller-manager-668cdd7dcf-7rp6f 2026-09-07T02:59:28Z Normal AddedInterface trainer-operator-controller-manager-668cdd7dcf-7rp6f Add eth0 [10.134.0.17/23] from ovn-kubernetes 2026-09-07T02:59:28Z Normal Pulling trainer-operator-controller-manager-668cdd7dcf-7rp6f Pulling image "quay.io/opendatahub/odh-trainer-operator:odh-stable" 2026-09-07T02:59:28Z Normal ScalingReplicaSet trainer-operator-controller-manager Scaled up replica set trainer-operator-controller-manager-668cdd7dcf from 0 to 1 2026-09-07T02:59:31Z Normal Pulled trainer-operator-controller-manager-668cdd7dcf-7rp6f Successfully pulled image "quay.io/opendatahub/odh-trainer-operator:odh-stable" in 2.6s (2.6s including waiting). Image size: 219739301 bytes. 2026-09-07T02:59:31Z Normal Started trainer-operator-controller-manager-668cdd7dcf-7rp6f Started container manager 2026-09-07T02:59:31Z Normal Created trainer-operator-controller-manager-668cdd7dcf-7rp6f Created container: manager 2026-09-07T03:00:39Z Normal ScalingReplicaSet trainer-operator-controller-manager Scaled up replica set trainer-operator-controller-manager-7f578b678f from 0 to 1 2026-09-07T03:00:39Z Normal AddedInterface trainer-operator-controller-manager-7f578b678f-wsz2n Add eth0 [10.133.0.21/23] from ovn-kubernetes 2026-09-07T03:00:39Z Normal Pulling trainer-operator-controller-manager-7f578b678f-wsz2n Pulling image "quay.io/opendatahub/odh-trainer-operator@sha256:835dfcd9e6bb439409c745d988fcc6ce0278077eb0585647681ab8d87aeb16ad" 2026-09-07T03:00:39Z Normal SuccessfulCreate trainer-operator-controller-manager-7f578b678f Created pod: trainer-operator-controller-manager-7f578b678f-wsz2n 2026-09-07T03:00:42Z Normal Pulled trainer-operator-controller-manager-7f578b678f-wsz2n Successfully pulled image "quay.io/opendatahub/odh-trainer-operator@sha256:835dfcd9e6bb439409c745d988fcc6ce0278077eb0585647681ab8d87aeb16ad" in 2.423s (2.423s including waiting). Image size: 219736229 bytes. 2026-09-07T03:00:42Z Normal Started trainer-operator-controller-manager-7f578b678f-wsz2n Started container manager 2026-09-07T03:00:42Z Normal Created trainer-operator-controller-manager-7f578b678f-wsz2n Created container: manager 2026-09-07T03:00:53Z Normal SuccessfulDelete trainer-operator-controller-manager-668cdd7dcf Deleted pod: trainer-operator-controller-manager-668cdd7dcf-7rp6f 2026-09-07T03:00:53Z Normal Killing trainer-operator-controller-manager-668cdd7dcf-7rp6f Stopping container manager 2026-09-07T03:00:53Z Normal ScalingReplicaSet trainer-operator-controller-manager Scaled down replica set trainer-operator-controller-manager-668cdd7dcf from 1 to 0 --- FAIL: TestPostUpgradeTrainerStillReady (300.05s) FAIL FAIL github.com/opendatahub-io/trainer-operator/test/e2e/upgrade/post 300.094s FAIL make: *** [Makefile:159: test-e2e-upgrade-post] Error 1